Self-Supervised Visual Acoustic Matching
Authors: Arjun Somayazulu
Changan Chen
Kristen Grauman
Venue: NeurIPS 2023 poster (2023)
Keywords: Audio-Visual learning
Visual Acoustic Matching
TL;DR: We propose a method for visual acoustic matching that trains a conditional audio GAN to disentangle acoustics from speech, jointly with a WaveNet-based acoustic matching model. The approach outperforms existing methods on in-the-wild environments.

Abstract: Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. Existing methods assume access to paired training data, where the audio is observed in both source and target environments, but this limits the diversity of training data or requires the use of simulated data or heuristics to create paired samples. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio---without acoustically mismatched source audio for reference. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric that quantifies the level of residual acoustic information in the de-biased audio. Training with either in-the-wild web data or simulated data, we demonstrate it outperforms the state-of-the-art on multiple challenging datasets and a wide variety of real-world audio and environments.

Self-Supervised Visual Acoustic Matching
Arjun Somayazulu' !
Changan Chen'
Kristen Grauman 1,2
'T Austin
FFARR, Meta
Abstract
Acoustic matching aims to re-synthesize audio clip sound if it were recorded in a target acoustic environment. Existing methods assume access to paired training data, where the audio is observed in both the source and target environments, but this limits the diversity of training data or requires the use simulated data heuristics create paired samples. We propose self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio- acoustically mismatched source audio for reference. Our approach jointly learns to disentangle room acoustics and re- synthesize audio into the target environment, via conditional GAN framework and a novel metric that quantifies the level of residual acoustic information in the de-biased audio. Training with either in-the-wile web data or simulated data, we demonstrate it outperforms the state-of tetee-of-hhe--as the-art on multiple challenging datasets and wide variety of real-world audio and environments. Project page: Intpss //vision. cs utexas.
vam
1 Introduction
The acoustic properties of the audio we hear are strongly influenced by the geometry of the room and the materials that make up its surfaces-large, empty rooms with hard surfaces like concrete and glass lead to longer reverberation times, while smaller, more cluttered rooms with soft materials like curtains and carpets will absorb sound waves quickly and produce audio that sounds dull and anechoic. Human perception exploits this acoustic- vriuua correspondence, and rely perceiving natural-sounding audio that is consistent with our environment as we navigate daily life.
Likewise, this phenomenon is important in virtual environments, such in AR/VR applications. When one hears audio that acoustically consistent with the virtual environment they are seeing, their brain can better integrate audio and visual information, leading to more immersive experience. Conversely, when one hears audio that does not match the expected acoustics of the virtual environ- ment, the perceptual mismatch can be jarring. The problem of audio-visual acoustic correspondence extends well beyond AR/VR applications. Film and media production involve recording audio in diverse spaces, which can be expensive and logistically challenging. Similarly, interior designers and architects face the problem of previewing how space will sound before it is built.
Today's 's approaches for modeling acoustic-visue coherence typically assume physical access to the which can be impractical impossible in some cases. particular, in acoustic matching, the audio captured in one environment is re-synthesized to sound as if it were recorded in another target environment, by matching the statistics (e.g. reverberation time) of audio samples recorded in that target environment [9] [5] 21 24] 26 47). Visual acoustic matching (VAM) instead takes an image of the target space as input, learning to transform the audio to match the likely acoustics in the depicted visual scene 33 (see Figure [a)). In both cases, however, learned models idcally have access to paired training data, where each training audio clip is recorded in two different environments. This permits straightforward supervised learning strategy, since model can leam to transform one clip (source) to the other (target). See Figure [bb, left. Unfortunately, this approach
target space 40 25
a)).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

b)
Paired
Unpaired
Source
Target
Sourc
Target
?
Figure I: Self-supervised visual acoustic matching, (a) Given source audio and target image the goal is to re-synthesize the audio to reflect the acoustics of the target environment. b) Two possible data settings: the paired audio setting (left) observes both the source audio and the audio in the target environment, allowing for supervised training. while the unpaired audio setting (right), observes only the audio in the target environment. Our self-supervised strategy handles the unpaired setting.
puts heavy demands on data collection: large-scale collection of paired data from a variety of diverse environments is typically impractical.
In-the-wild web videos provide us with large, readily available corpus of diverse acoustic environ- ments and human speakers. However, this data is umpaired-we only observe the sound recorded in the target space, without a second recording in a different environment for reference. Prior work attempts to turn unpaired data into paired data by using an off-the-shelf dereverberator model 7 to produce pseudo-anechon passed to a visual acoustic matching ( (VAM) model with the true reverberant audio as the target [3]. While that approach is inspiring, it has a fundamental limitation. The automatic dereverberation process is necessarily imperfeet, which means that residual acoustic information indicative of the target environment's 's acoustics can remain in the (pseudo) source example. In turn, the acoustic matching model trained produce the target audio can learn to use those residual acoustic cues in the indio- of the target space's s visual features. When evaluated at test-time on arbitrary source audio and unseen images, the residual acoustic clues exploited during training no longer available, leading to poor acoustically matched audio outputs.
source recordings from in-the-wild reverberant audio, which are then
We propose self-supervised visual acoustic matching approach that accommodates training with unpaired data from in-the-wild videos (See Figure[]b), right). Our key insight is a training objective that explicitly removes residual acoustics in the audio, forcing reliance the visual target. In bebbasser-wichh is trained to adversarially
particular, our approach jointly trains an audio-visual minimize residual acoustic information in the dereverberated audio allogside reverberator that performs visual acoustic matching. To this end, we develop an acoustic residue metric that quantifies the level of residual acoustic information in a waveform, based the difference in performance between a) an acoustic matching model that conditions on the target image and b) a model that does not condition on any image. Intuitively, training on audio with low residual acoustic information frees the model from relying on that (unrealistic) information during training, allowing it to instead leverage the necessary visual cues.
We use a time-domain conditional GAN as our debiaser, and continually update the integrated reverberator the distribution of generated audio shifts during training. Unlike prior work, our approach allows training directly on unpaired vidcos and speech
Our proposed LeMARA model outperforms existing approaches 7 14 on challenging in-the-wild audio and environments from multiple datasets. Further, to benchmark this task, we introduce a high audio-visual correspondence subset of the AVSpeech [10] video dataset. Though we focus on the task of visual-acoustic matching, our insight potentially has broader implications for other self-supervised multi-modal learning tasks in which one wants to control the impact of a dominating modality.
2 Related work
Room acoustics and spatial sound Audio-visual acoustic matching has limited prior work, though there is growing interest from vision and audio communities. Image2Reverb [40] learns to map an input image to its corresponding Room Impulse Response (RIR) the transfer function that
https://rulonce hatprr/lono..cooffaaaaeeeeeeeeee
modern- auditori -orsiiddddiiim//
We focus on clean human speech in indoor settings given its relevance to many of the applications cited above, and due to the fact that human listeners have strong prior knowledge about how room acoustics should affect speech. However, our model design is not specific speech and could be applied other audio sources.

---

characterizes acoustics at particular listener/speaker location which can then be convolved with an arbitrary source waveform. In audio-only approaches, 41 32 331 generate RIRs from reverberant audio clips, and [42] uses WaveNet model to warp source audio into a target space conditioned on learned embedding of the target audio. Generalizing RIR inference to full 3D environments, Few-Shot RIR (25) and Neural Acoustic Fields [23] sample impulse responses at multiple places in an environment in order to synthesize the RIR at novel locations with a transformer. In related tasks, Novel View Acoustic Synthesis [5] directly synthesizes a source audio at new camera pose in the room, while other methods binauralize monaural source audio [6 36]. Unlike our model, which train from arbitrary images, existing methods require knowledge of the ground truth RIRs [41 32,
40 23 25] or paired source-target audio [5] 16 36.. Most relevant to our approach, AVITAR 3 uses cross-modal transformer to re-synthesize the audio; it relies an off-the-shelf audio-visual dereverberator trained in simulation (7] to produce a pseudo-clean source signal, and suffers from the acoustic residue issue discussed above. Our results illustrate how our model overcomes this shortcoming.
33
Speech synthesis and enhancement Recent work for speech re-synthesis treats acoustic properties as a "style" which can be disentangled from underlying speech and used to perform acoustic-style matching of audio [27 31 30 46). However, they require either paired data or speaker embeddings to be learned. Web videos (our target domain) can consist of entirely unique speakers, making it difficult to learn a robust speaker embedding. While environment-awant
text-to-speech 43 is applicable even in settings where each speaker in the dataset is unique, the model requires access to paired speech for supervised training. Supervised methods for speech enhancement and dereverberation assume access to paired training data, i.c., anechoic "clean" reference waveforms alongside the altered
waveforms [4 7 48, 37, 12 [3]. Unsupervised speech enhancement approaches [22] 19 14 such
as MetrieGAN-U
optimize generic perceptual speech quality scores, such as PESQ 38 (which
requires paired samples) and speech-to-revene roreerrbraran modulation energy (SRMR) [1
(which does
not). While we share the objective of relaxing supervision requirements, our overall goal is distinct: rather than optimize a generic quality metric, we aim to retarget the input sound to match a specific (pictured) environment. To achieve that goal, we introduce a novel debiasing objective applicable in a conditional GAN framework.
3 Approach
We present Learning to Match Acoustics by Removing Acoustics, LeMARA, to achieve self- supervised visual acoustic matching (VAM). Let A E A denote an audio waveform and let V E denote image frame. During training, we given N unlabeled examples of target audio and scenes {AA, Vo)l taken from frames and accompanying audio in YouTube videos (cf. Sec. for dataset details). While the data is multi-modal, it is unpaired: it has both audio and visual features, but it lacks a paired sample of the audio in some other source domain (see Fig. [ With this data, we wish to learn function fA,, Vi)= A. V A that takes source audio A, (which may may not be reverberant) and a target image Ve, and produces the audio re-synthesized to sound like it was recorded in the target scene.
To self-supervise / using unpaired training data, we would like a model that (1) dereverberates A, to strip it of its room acoustics, yielding pseudo- source audio A(()) and then (2) reverberates A(*) by "adding" in the room acoustics of image V2 to regenerate the target audio: f(A(*) V)) ~ A, A naive collapse to a trivial solution of f doing nothing. better solution would pre-train the dereverberator with well- audio-only model, yet as we show in experiments, this leaves residuals of the target environment in the audio A(t) that handicap the reverberator at inference time, when such signals will not exist.
joint training of two such modules-a dereverberator and reverberator-
The key insight of our approach is to make the dereverberation stage de-biasing stage that is explicitly trained to remove room acoustics information not available in the target image V,. This forces reliance on the visual target and helps our model f generalize to unseen audio and scenes.
Our model has two main components that are trained jointly: (1) de-biaser model G responsible for removing room acoustics from the target audio and (2) a visually-guided reverberator that injects acoustics from the target environment into the output waveform. We first introduce the de-biaser architecture (Sec. 3.T,, followed by our novel acoustic residue metrie that is optimized during de- training (Sec. 3.2), and our training strategy that allows joint fine-tuning of the
3

---

reverberator and de-biaser (Sec. B.3). Finally, we present our approach for training self- VAM (Sec. 3.4). TigureBovervee our model.
3.1
De-biasing Conditional Generative Adversarial Network
The role of our de-biaser is dereverberate audio samples in a way that minimizes any residual room acoustics information. To that end, we devise an adversarial de-biaser based on MetricGANs [12
[4]. MetrioGANs are similar to conventional generative adversarial networks (GAN) [17 with a generator that aims to enhance a speech waveform-exee exceep the discriminator's job is to mimic the behavior of some target quality function. Our de-biaser module extends the basic MetricGAN, augmenting it with novel acoustic residue metric (Sec. (Sec.3.2) and training procedure (Sec. 3.3 that
accounts for the evolving distribution of the de- viebee audio.
Our GAN consists of a generator G. a discriminator D, and an audio quality metric M. D is trained to approximate the quality metric M,, and G is trained to maximize the metric score on generated data, using D as a learned surrogate of the metric function M. G is a conditional generator: given an input waveform, it produces modified waveform which optimizes the quality metric score.
Let {As, } be a dataset of target audio-image pairs, and let M(A,, V/) E [0, be our quality metric M (defined below) that produces a scalar measure of the residual room acoustic information in audio. As in the conventional GAN framework, we alternate between discriminator and generator updates. During an epoch of discriminator training. D trains to approximate the metric function M's scores on both raw audio A₁ and generated audio G(Ar). The discriminator loss function is:
LD I M(A,, Volll2 + IDD(G(A)) - MMG((A) Vi)l2 + (ID(An)))) - Shisella (I)
where the first and second terms incentivize D to produce score estimates that approximate the metric function when evaluated on raw input audio (At) and generated audio G(A,), respectively. Following 112, the third term trains the discriminator {(Aniet, Shist)), where Ahist = (At) is a generated sample from a previous epoch, and Shiet = Mpreo(Gpred (At), Vi) is its associated metric score. Training on these historical samples helps improve stability and mitigates issues with catastrophic forgetting in the discriminator.
samples from historical replay buffer
The generator is trained with an adversarial loss, using the discriminator D) learned from the previous epoch of discriminator training (which depends only on A,) as a surrogate of the true metric M (which depends on both A, and V,). The generator loss is
LG = ()t)) - 1/
(2)
Our metric is normalized to produce scores between 0 and 1 being optimal), so this loss forces G to generate audio that maximizes the estimated metric score. Next, we introduce our metric M.
3.2 Acoustic Residue Metric
Rather than train the de-biaser GAN to optimize a generic speech quality metric 12 14], we wish to quantify the amount of residual room acoustics information in audio sample. Hence, we define metric M that allows the downstream reverberator model R itself to quantify the level of residual acoustic information in the waveform. Specifically, the metric consists of two models trained to perform reverberation on dereverberated speech. Importantly, one model RG has been trained to perform VAM (using the target image as conditioner), while the other, Rgg has been trained perform blind acoustic matching (without the target image as conditioner). We next define the reverberator modules, and then return their role in M.
Inspired by recent work in time-domain signal modeling 45 36 5] we use a WaveNer-like architec- ture for the reverberators consisting of multiple stacked blocks of 1D convolutional layers, with an optional gated fusion network to inject visual information for Rg, Similar to [3], the reverberators use sinusoidal activation function followed by two separate 1D cony layers that produce residual and skip connections, the latter being mean pooled and fed to decoder to produce reverberated audio. We choose this model because it is parameter-efficien consisting entirely of ID convolutions, and because it allows time-domain conditional waveform generation. See Sec. 5 for training details.
We these models to compute the acoustic residue metric. Given input audio A and image V, our metric is defined as:
|R (R((R(())) - RT(A.)) - (R, RT(AII)
M(A, V) -
(3)
max (T.1,

---

where A, denotes the known target audio and RT is a scalar-output function charac- terizing the general reverherant properties of its input audio, which we define below.
Reverberant
Denmeebarated ig
De biased
Wavefomm
Eqn. B quantifies the level of acoustic residue-thal is, how much greater the blind reverberator's error is compared to the iisually-guides reverberator's 's error. If de-biasing of A has gone well, this value will be high. When evaluated on audio that contains a high level of residual acoustic information, the visual features will not provide additional useful information, re- sulting similar performance by both vi- sual and blind reverberation models. In other words, if the two errors are similar, vi- sual is not adding much, and there is room acoustic information lingering in the audio A. This pushes the M score to be smaller (poor quality under the metric). On the other hand, when a waveform contains very little residual acoustic information, the visual features will help the visual model R, produce a more accurate acoustically matched waveform model with lower error compared to the blind model Rg. This will result in higher metric score.
Figure 2: De-biased audio. The de-biaser removes residual acoustic traces in audio, and attenuates long sound decay faster than in dereverherated speech.
Figure demonstrates the effect of de-biasing. Reverberant audio (left) has been imperfectly dereverberated (middle), leaving residual reverberation trails which contain information about the original acoustic environment, De-biased audio (right) removes these residual artifacts. The waveform plots (right) show de-biased audio (green) significantly attenuates the long sound decay present in both reverberant (blue) and dereverberated (orange) audio. This forces the reverberator to learn acoustics from the target image.
For the function RT in Eqn. we leverage a classic content-invariant metric for characterizing room acousties: the Reverberation Time at 60 or "RT60". RT60 is the amount of time after a sound source ceases that it takes for the sound pressure level to reduce by 60 dihhh depends directly on the geometry and materials of the space. For example, no matter the initial direct sound, big cathedral will yield a high RT60, and a cozy living room will yield a low RT60, While RT60 can be quantified by sensing when one has access to the physical environment, we use a learned estimator for RT60 to allow generalization (see Sec. for details). In Eqn. 3 the normalization by the RT60 of the target audio improves stability of discriminator training. We use the clipping function max maz(0.1.) here to prevent samples with extremely low RT60 from destabilizing training.
Training with this acoustic residue metric allows the downstream reverberator models R. Rs them- selves to improve the performance of the de-biaser model G. Unlike SRMR, DNSMOS [35] or any existing off-the-shelf metric that quantifies general speech quality, our metric directly addresses the problem of residual acoustic information audio. Although G may learn a function similar that of dereverberation, we use the term de-biaser to describe the generator to highlight the novel training objective it is trained against, which distinguishes it from a conventional dereverberation model.
3.3 Joint Training of the De-biaser and Reverberators
At initialization, R, and Rs trained certain distribution of speech. When training the de-biasing GAN with the acoustic residue metric, generated speech can eventually fall out of the distribution on which these reverberators were trained, causing M to produce unreliable metric scores that destabilize training. To address this, we introduce a strategy to update R2 and Rs during training of the de- biasing GAN. Updating these models ensures that M consistently produces reliable acoustic residue scores as the distribution of generated speech shifts over the course GAN training.
At the start of GAN training we initialize the target networks R', and RÝ, as copies of R, and Rs respectively. During discriminator training, each batch of (A,,), V.)) samples are passed to R, and Rg to compute metric scores under their current frozen state. This batch is also passed to the
5

---

a)
Reverbernnt Audio Ar *tallf fpix
Deneverbentor
De-binser (G)
Reverberatar (R2)
A2
b)
Metric (M)
V
M(4)
( Reverberant or eleaned andio
MSE Loss D(A)
RGB Image
Discriminator (D)
Altemately updated
Vt
Altematelly updated
R
Ap
A
Geueratox (G)
Discriminator (!)
Loss
+ (+
M(A)
Derovenberated
Audio
mndio
D(G(A))
Target
R
Ap
A
MSE
Score
Loss
Loss
Figure 3: LeMARA overview. a) Training procedure. Reverberant audio is first processed with an off-the-shelf dereverberator. is then input to a de-biaser model which strips acoustics from the audio. The clean audio is passed to the reverberator along with the target image for acoustic matching. b) De-biaser architecture. G is trained to adversarially maximize the score of the discriminator D, which learns surrogate of the acoustic residue metric M (Sec. [3. and 3.2) c) Acoustic residue metric. Both R2 and RG are continually trained on generated data to ensure that they provide accurate
metric scores as the distribution of generated data evolves during training (Sec. 3.3 At test time, we use the trained de-biaser G and the visual reverberator R, to perform VAM.
target networks, which compute the losses
Lvisual
())))()),))) All2
(4)
Lolind
[G()) - All2
(5)
and perform an update step. Every E epochs, the target networks' weights are copied over into the metric networks. This update strategy allows R, and R2 to be jointly fine-tuned with the de-biaser model G. Figure overviews the model components and data flow.
3.4 Training and Inference
Training proceeds in three steps. (1) First we pretrain the de-biaser G. This entails pretraining a MetrieGAN-U [44] with the tea--to--neggy-MModee
Ratio (SRMR) metric [11] on speech pre-processed with our off-the-shelf dereverberator (see Sec. 5). By refining the dereverberated output with the MetricGAN-U,, we improve its quality and intelligibility without introducing ad- ditional supervision requirements. (2) Second, we pretrain the reverberators that perform (visual) acoustic matching. Specifically, we train R2 and Rd with Luisnal and Laiind, respectively, using the dereverberated and SRMR-optimized outputs from the MetricGAN in step (1). (3) Finally, we jointly fine-tune both the di-biaser G and reverberators, using the acoustic residue metric (Eqn. 3 for the GAN metric M, the generator and discriminator losses given in Eqns. 2 and || together with our alternating training scheme defined in Sec. 3.3 To improve stability in training. since the discriminator /) starts with a good approximation of the SRMR metric, we continue training in step 3 using a weighted combination of SRMR and our residue metrie: aSRMR A) + (1 - a)M( V). Although the acoustic residue is approximately differentiable, the implementation of SRMR we use not, motivating our use MetricGAN over an end-to-end alternative. This approach also allows for extensibility to other non-differentiable speech quality scores such PESQ.
At test time, use LeMARA for target environment image (7()) we apply the trained de-biaser G followed by the visual reverberator R,:
matching follows: given a source audio A(2) and
((A
v(()) R2, (G(A!)) V,("))
(6)
Altogether, our approach adds the room acoustics depicted v((1) to the input audio. the case that the source audio A(a) is known to be anechoic (e.g., user is using LeMARA to re-synthesize stock anechoic sounds for a new scene), then we simply bypass the de-biaser G and directly apply R.,

---

Low reverh
High reverb
Figure 4: Datasets. SoundSpaces-Speech (top row) renders panoramic views of people in indoor environments. AVSpeech-Rooe Rooom (bottom) contains wide variety of naturalistic indoor environments with diverse acoustic properties.
4
Datasets
We use two datasets: SoundSpaces-Speech 7 and AVSpeech [10] See Figure 4 For all results, we test only on audio and environments that are not observed during training.
is audio-visual dataset created using the SoundSpaces audio simulation platform [4 together with 82 3D home scans from Matterport3D [2] and anechoic speech samples from LibriSpeech [28]. SoundSpaces offers state-of-the-arr genera- tion of Room Impulse Responses (RIRs) computed at regular source-listener locations throughout the environment, accounting for major real-world acoustic phenomena (reverberation, early specular/diffuse reflections, etc.) See 7 for details.
SoundSpaces-Speedt
SoundSpaces-Speech
By convolving anechoic sample with the appropriate RIR, it allows realistic simulation of audio reverberation at arbitrary locations in the pre-scanned real-world Matterport environments. A 3D humanoid is rendered the source location in the Matterport home. SoundSpaces-Spert Speeeh consists of anechoic speech samples from LibriSpeech paired with their acoustically-comees reverberated waveform (rendered using SoundSpaces) any 82 unique environments, together with an RGB- D image at the listener's position. To train on SoundSpaces- we discard the source audio and only use the target audio (simulated reverberant audio). We train/val/test splits of 28,853/280/1. ,489 samples.
in self-supervised setting,
AVSpeech 10] is a large-scale video dataset consisting of 3-10 second clips from YouTube videos, most of which feature single speaker and little background noise. Not all of the clips have naturalistic audio-visual correpondence, due to video editing tricks, microphone locations, virtual backgrounds, etc. Hence, we create subset of AVSpeech, called AVSpeech- Rooms, that preserves samples with useful information about room geometry and material types (see Supp. for details). A randomly selected frame from the video clip is used the target image. Our final set consists of 72,615/1,91 1/1,91 train/val/test samples. See Figure 4 (bottom).
AVSpeech-Roome
5 Experiments
Implementation Details We use the procedure outlined in Sec. 3.4 to train SoundSpaces- Speech. For training on AVSpeech-Rooms, we proceed directly to step (2), pre-training R, and RG on audio that has been de-biased using the fine-tuned SoundSpaces-Speecl de-biaser model (instead of an SRMRRoptimized model trained on AVSpeech-Roone (3) in SoundSpaces-Specee the SoaddSaaes-Sppeed trained de-biaser to bypass step (1) when training on AVSpeech- -SSoecch-Rooms. While AVSpeech can be trained with the full procedure outlined in Sec. 3.4 (see ablations in Supp.), shortcut training allows us the advantage of the strong prior for de-biasing learned by the fine-tuned SoundSpaces de-biaser.
We then proceed to step use of
We refer to this setup as "shorteut training" highlight
train LeMARA using the combined acoustic residue metric with a = 0.7. We train WaveNet- based dereverberator (6) ("off-the-shelf") on paired SoundSpaces-Se paces-Speecc audio which is "reversed" (reverberant input audio, anechoic target audio). We use this dereverberator for both LeMARA and the ViGAS (5) baseline. Prior VAM work (3] used an audio-visual dereverberator trained on both
7

---

Table 1: VAM results two datasets (unseen environments). RTE standard errors less than 0.005, and STFT standard errors are 0.031 and 0.64 for the two datasets, respectively.
Train Test
SoundSpaces-
AVSpeech-Roome AVSpeech- Rooos
SoundSpaces- Speeech
LibriSpeech RTE (s)
Model
RTE (s)
STFT
logSTFT
RTE (s)
STFT
logSTFT
Input audio AVITAR
0.320
1,427
1.274
0.310
1.327
2.107
0.358
0.080
2,471
1.629
0.136
2.894
1.290
0.239
ViGAS
0.108
4.373
1.232
0.109
7.007
0.662
0.254
LeMARA(no vis)
0.152
5.612
1.884
0.137
6,256
0.705
0.223
LeMARA (ours)
0.079
0.690
1.530
0.071
6.298
0.571
0.210
simulated and real-world data to pre-process reverberant audio. For fair evaluation, we train their model with their same original uudio-visual dereverberator)
We adapt our code for the reverberator models and ViGAS from same hyperparameters and loss our reverberators during pre-training. Our de-biaser is adapted from the speechbrain MetricGAN-U implementation 34]. Our RT60 estimator is trained on pairs of reverberant samples from SoTA audio simulator [6] paired with the ground truth RT60 for the
36 We train ViGAS with the
RIR used to produce the reverberant sample. We use Resnet18 (18) model to encode our visual features on RGB images. The last feature map before the final pooling layer is flattened and used as the visual feature conditioner. See Supp. for training details and architecture for these models. We plan to release our code to facilitate further research.
Metrics We rely three metries to evaluate quality of VAM models. STFT Error and logSTFT Error compute the MSE loss and log MSE loss, respectively, between the magnitude spectrograms of predicted and target speech. Because we do not have access to the ground truth RIR, we also use an RIR reverberation metric that can be reliably estimated from audio, RT60 Error (RTE), which measures the MSE between RT60 estimates of predicted and target speech. The first two apply only when we have ground truth re-synthesized audio (in simulation), while the third is content-invarian and captures room acoustics signatures for any target.
Seen and unseen evaluation We report results on two different settings: unseen environments, where both the source audio A, and target image V2 come from the test set; and seen environments, where an audio sample Ag from the test set is paired with a target image V/ from the train set (seen by the model). The unseen environment setting important for evaluating generalization novel scenes (e.g. matching an audio sample to an image from the Web), while the seen environment setting is useful for cases which we already have video recordings, such as film dubbing.
Baselines As baselines, we compare to state-of-the-art models for audio-visse visual re-targeting: (1) AVITAR 33 the only prior method that addresses the visual-acoustic matching task. It consists of a Transformer for audio-visual fusion, followed by generator that synthesizes the reverberant waveform from the audio-visual latent feature. As discussed above, for self-supervised training, the authors use a pre-trained audio-visual dereverberation model 7 to create pseudo-source audio, which is passed as input to the model. (2) ViGAS (5), model designed for novel-view acoustic synthesis, camera pose. We adopt its Acoustic Synthesis module, WaveNet model based on (36], for our task. To apply it for VAM, we replace the camera pose with the flattened feature from the ResNet. (3) Non-visual LeMARA. We evaluate LeMARA with the blind reverberator Rg (4) Input audio. We copy the dereverberated audio to the output. AVITAR and ViGAS are trained with the data augmentation strategy introduced in 3 (see Supp. for details).
conditioned
fine-tuned during training.
Results on SoundSpaces-Speett columns) for the unseen and seen settings, respectively. LeMARA outperforms the baselines most metrics. Non-visual LeMARA performs significantly worse, indicating LeMARA effectively utilizes visual features during acoustic matching. This demonstrates the success of our acoustic de-biasing, which forces stronger learning from the visual stream. Notably, our model significantly outperforms
Tables and report results on Seaaddpaaes-SSppeee (left three

---

Table 2: Results seen environments. We evaluate VAM using audio samples from the test set paired with target images from the train set.
SoundSpaces-Spee -prrech
AVSpeech-Roome STFT
Model
RTE (s)
STFT
logSTFT
RTE (s)
logSTFT
AVITAR ViGAS
0.062
3.186
1.506
0.131
2.852
1.273
0.076
6.386
0.962
0.111
6.946
0.657
LeMARA (ours)
0.060
0.667
1.417
0.067
6.198
0.570
Normalized RTE vs Target RT60
RT60 Distribution
AVIEA
Source (LibriSpacch)
VXGAS
12
AVIAAR
1eHARA
VIGAS
1.0
LeMARA
Normalized RTE
0.8
Target
0.8
RT60
0.4
0.4
02:
00
0.8
Target RT60
Figure 5: Normalized RTE across reverberation levels (left). We evaluate RTE normalized by target RT60 on AVSpeech- Rooms samples with varying levels of reverberation. LeMARA consistently achieves lowest error at all reverberation levels. RT60 distribution (right). The distribution of RT60 values for audio generated by LeMARA best matches the target distribution.
reverberator -indicating that our performance improvement over AVITAR can be attributed to our novel training objective, and not simply due to a shift in architecture from Transformer WaveNet.
ViGAS-whiek shares the same architecture
Results AVSpeech-Rooms Tables 1 and 2 show results AVSpeech-Ron
(right four
columns) for both unseen and seen settings, respectively. We further divide the unseen evaluation into two scenarios: (1) Where the source audio and target visual come from the same (unobserved) AVSpeech-Ron Rooms sample and (2) where the source audio comes from LibriSpeech [29], dataset of anechoic source samples of people reading passages English (rightmost column). In both cases the target visual environment is a frame from an unseen AVSpeech video.
LeMARA outperforms the basclines in both settings RTE. We outperform ViGAS in the Lib- riSpeech scenario despite sharing the same reverberator architecture, highlighting the efficacy of our novel training objective. Figure 3 (right) shows the distribution of RT60 values for audio reverberated by our model (blue), the baselines, and the target RT60 distribution (purple). Our model best matches the target RT60 distribution. LeMARA also outperforms baselines across samples with diverse range of real-world reverberation intensities. We stratify the test set by ground truth RT60 and evalu- ate performance samples within each reverberation bin (Figure 5 left). LeMARA outperforms the baselines across all reverberation levels, demonstrating its robustness to variation real-world data.
Although our model performs poorly STFT error, significantly outperform existing approaches on logSTFT error, which better reflects perceptual quality given the logarithmic nature of human hearing. Furthermore, the naive baseline achieves the lowest STFT error, indicating that derever- berated audio itself strongly resembles reverberant audio even prior to acoustic matching. Models that use this dereverberator without further de-biasing will display artificially low STFT error when evaluated in-dataset (AV S.VVSpeech-oooe in-dataset evaluation with the LibriSpeech generalization case (far right in Table I where soundly outperform Input audio) to gain a complete picture of model performance.
- AV Seneech-RRomss Thus, is important to balance the
Furthermore, "'Speech-Rooms contains a variety of non-speech sounds (e.g. air conditioning, clicking/tapping noises from object interactions) which make reverberation even more challenging. These signals may be perceptually weak, but they appear spectrograms and contribute the larger

---

Target XT.00 0.1638
AVIARR
LeMARA
Source
Target Image
LeMARA
RXTA0 - 0.01 77
DAP222
0,1619
RTSO 0,0256
RT60 0.3094
RT500 -0.1054
RT60 0.3748
RT660 - 0.008]
RT66 0.3698
0.1222
RT60 0808
RI100 0 0.0699
0.5129
0.119
RT
Figure 6: Acoustically matched audio. LeMARA produces audio with more accurate acoustics than AVITAR across diverse acoustic and visual scenes. Shown here for LibriSpeech setting.
STFT error we observe on AVSpeech- Rooms. In contrast, SoundSpaces-Speece contains no artifacts or background noise that could contribute to spectrogram errors.
An ablation study of our model analyzing the impact of varying the target metric during training shows the clear advantage of our residue metric compared to SRMR alone (see Supp.). In particular, training with an SRMR objective alone yields an RTE of 0.2308 on LibriSpeech, whereas training with pure acoustic residue metric yields an RTE of 0.2156; combined metric yields RTE of 0.2123. This reinforces the efficacy of our metric as a training objective: de-biasing is distinct from generic enhancement.
Figure shows examples of our model's generated sounds for different target images, compared to the output of AVITAR [] (the best performing baseline). We display the RT60 of the source audio, the visual scene's s ground truth RT60, and the RT60 the two method's ' generated audio. In the majority of cases, scene. Our model learns to add little reverberation in enclosed environments with soft surfaces (top left), and to add more reverberation in open acoustic environments with hard surfaces (bottom right). We also highlight a failure mode in which our model does not inject proper acoustics (bottom left), likely due the irregular shape the room. See our project webpage listen examples.
model produces audio that more closely matches the true acoustics of the visual
Human perception study We augment these quantitative results with a human subject study, in order to gauge whether listeners perceive our results as successfully retargeting the audio to the pictured environment. We design survey using 23 images Vi(0) selected from -VSpeech-Roome that show room geometry and materials clearly, and are representative of a diverse variety of acoustic environments. We couple those with 23 anechoic source samples A! from LibriSpeech (29] For each sample, we generate the acoustically matched audio with both LeMARA and AVITAR best performing baseline. We anonymize and shuffle the generated audio, and ask 10 subjects with normal hearing identify which room matches best with the audio among three given rooms, one
the
of which is the true room v,(*), Users correctly identified the target room with 46. 1% accuracy on speech generated by LeMARA versus 34.7% accuracy with speech generated by AVITAR. This shows our model achieves higher quality acoustic matching according to human perception. That said, the subjects' accuracy rates are fairly low in the absolute sense, which suggests both the difficulty of the problem and subtlety of the perception task. Our results have pushed the state of the art, both objectively and subjectively, but there remain challenges fully mature visual acoustic matching.
6 Conclusion
We introduced self-supervised approach to visual acoustic matching. Built on novel idea for disentangling room acoustics from audio with a GAN-debiaser, our model improves the state of the art on two datasets. Our acoustic residue metric and adversarial training has potential to generalize to other multi-modal learning settings where there is risk of unintentionally silencing a paired modality during training. For example, our framework could be explored for binauralization of mono sounds using video audio-visual source separation. In future work, we plan to explore generalizations to spatial dynamics that would account for movement of speaker throughout 3D space.
10

---

Acknowledgments: Thanks to Ami Baid for help in data collection. UT Austin is supported in part by the IFML NSF Al Institute. K.G. is paid research scientist at Meta.
References
[1] Deepak Baby and Sarah Verhulst. Sergan: Speech enhancement using relativistic generative ad- versarial networks with gradient penalty. ICASSP 2019 2019 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 106-110, 2019.
[2] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV, 2017.
[3] Changan Chen, Ruohan Gao, Paul Calamia, and Kristen Grauman. Visual acoustic matching. In CVPR, 2022.
[4] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. SoundSpaces: Audio- visuull navigation in 3d environments. In ECCV, 2020.
[5] Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. In CVPR, 2023.
[6] Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philipp Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-ccoustic learning. In NeurlPS, 2022.
[7] Changan Chen, Wei Sun, David Harwath, and Kristen Grauman. Learning audio-visual derever- beration. In ICASSP, 2023.
[8] Sasha Devore and Bertrand Delguite. Effects of Reverberation the Directional Sensitivity of Auditory Neurons across the Tonotopic Axis: Influences of Interaural Time and Level Differ- ences. Journal of Neuroscience, 30123):7826-7837 2010. Publisher: Society for Neuroscience _eprint: https://wwwwjneeeeee
[9] James Eaton, Nikolay Gaubitch, Allistair Moore, and Patrick Naylor. Estimation of room acoustic parameters: The ACE challenge. /EEE/ACM Transactions on Audio, Speech, and Language Processing, (441)), 2016.
[10] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein. Looking to listen at the cocktail party. ACM Transactions on Graphics, 37(4):1-11, jul 2018.
[11] Tiago H. Falk, Chenxi Zheng, and Wai- Yip Chan. non-intrusive quality and intelligibility measure of reverberant and dereverberated speech. IEEE Transactions on Audio, Speech, and Language Processing, 18(7):1766-177) 1674 2010.
[12] Szu- Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. MetricGAN: Generative adversarial networks based black-box metric scores optimization for speech enhancement. In ICML, 2019.
[13] Szu-Wei Fu, Cheng Yu, Tsun-An Hsieh, Peter Plantinga, Mirco Ravanelli, Xugang Lu, and Yu Tsao. Metricgant: An improved version of metricgan for speech enhancement. In Inter- Speech, 2021.
[14] Szu-Wei Fu, Cheng Yu, Kuo-Hsuan Hung, Mirco Ravanelli, and Yu Tsao. Metricgan-u: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speech. In arxiv, 2021.
[15] Hannes Gamper and Ivan J Tashev. Blind reverberation time estimation using convolu- tional neural network. In 2018 16th International Workshop on Acoustic Signal Enhancement (IWAENC), pages 136-140, 2018
[16] Ruohan Gao and Kristen Grauman. 2.5d visual sound. In CVPR, 2019.

---

[17] Ian J. Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In NeurIPS, 2014.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[19] Bryce Irvin, Marko Stamenovic, Mikolaj Kegler, and Li-Chia Yang. Self-supervised learning for speech enhancement through synthesis. In arxiv, 2022.
[20] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution region supervision, 2021.
[21] Florian Klein, Annika Neidhardt, and Marius Seipel. Real-time estimation of reverberation time for selection of suitable binaural room impulse responses. In Audio for Virtual, Augmented and Mixed Realities: Proceedings of 5th International Conference Spatial Audio (ICSA 2019),
pages 145-150, 2019.
[22] Hsin- Yi Lin, Huan-Hsin Tseng, Xugang Lu, and Yu Tsao. Unsupervised noise adaptive speech enhancement by discriminatoor-constrriin optimal transport. In NeurlPS, 2021.
[23] Andrew Luo, Yilun Du, Michael J. Tarr, Joshua B. Tenenbaum, Antonio Torralba, and Chuang
Gan. Learning neural acoustic fields. In NeurIPS, 2022.
[24] Wolfgang Mack, Shuwen Deng, and Emanuël AP Habets. Single-channel blind direct-to- reverberation ratio estimation using masking. In INTERSPEECH, pages 5066-5070, 2020.
[25] Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kristen Grauman. Few-shot audio-visun visual learning of environment acoustics. In NeurlPS, 2022.
[26] Prateek Murgai, Mark Rau, and Jean-Mare Jot. Blind estimation of the reverberation fingerprint of unknown acoustic environments. In Audio Engineering Society Convention 143. Audio Engineering Society, 2017.
[27] Ahmed Omran, Neil Zeghidour, Zalán Borsos, Félix de Chaumont Quitry, Malcolm Slaney, and Marco Tagliasacchi. Disentangling speech from surroundings in neural audio codec. In arxiv, 2022.
[28] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210, 2015.
[29] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[30] Adam Polyak. Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. In InterSpeech, 2021.
[31] Kaizhi Qian, Yang Zhang, Shiyu Chang, David Cox, and Mark Johnson Unsuper- vised speech decomposition via triple information bottleneck. In arxiv, 2021.
[32] Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. IR-GAN: Room Impulse Response Generator for Fir-Field Speech Recognition. In Proc. Interspeech 2021, pages 286-290, 2021.
[33] Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. Ts-rir: Translated synthetic room impulse responses for speech augmentation. 2021 IEFE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 259-266, 2021.
[34] Mirco Ravanelli, Titouan Parcollet, Aku Rouhe, Peter Plantinga, Elena Rastorgueva, Loren Lugosch, Nauman Dawalatabad, Chou Ju-Chieh, Abdel Heba, Francois Grondin, William Aris, Chien-Feng Liao, Samuele Cornell, Sung-Lin Yeh, Hwidong Na, Yan Gao, Szu-Wei Fu, Cem Subakan, Renato Mori, and Yoshua Bengio. Speechbrain. https: ///g/t//tttt thub. com/ speecabraiin/gpeechbraii 2021.
12

---

[35] Chandan K A Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric evaluate noise suppressors. In arxiv, 2021.
[36] Alexander Richard, Dejan Markovic, Israel D. Gebru, Steven Krenn, Gladstone Alexander Butler, Fernando Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. In International Conference on Learning Representations, 2021.
[37] Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. In arxiv, 2022.
[38] Antony W. Rix, John G. Beerends, Mike Hollier, and Andries P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.012H37 2:749 752 vol.2, 2001.
[39] Manfred R. Schroeder. New method of measuring reverberation time. In The Journal of the Acoustical Society of America 37, 409, 1965.
[40] Nikhil Singh, Jeff Mentch, Jeny Ng. Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-modal reverb impulse response synthesis. In ICCV, 2021.
[41] Christian J. Steinmetz, Vamsi Krishna Ithapu, and Paul Calamia. Filtered noise shaping for time
domain room impulse response estimation from reverberant speech. In 202/ IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 221-225, 2021.
[42] Jiaqi Su, Zeyu Jin, and Adam Finkelstein. Acoustic matching by embedding impulse re- sponses. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 426-430. IEEE, 2020.
[43] Daxin Tan, Guangyan Zhang, and Tan Lee. Environment aware text-to-speech synthesis. In InterSpeech, 2022.
[44] Ke Tan, Yong Xu, Shi-Xiong Zhang, Meng Yu, and Dong Yu. Audio-visual speech separation and dereverberation with two-stage multimodal network. IEEE Journal of Selected Topics in Signal Processing, 14(3):542-553) mar 2020.
[45] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In arXiv, 2016.
[46] Zhichao Wang, Qicong Xie, Tao Li, Hongqiang Du, Lei Xie, Pengcheng Zhu, and Mengxiao Bi. One-shot voice conversion for style transfer based on speaker adaptation. In ICASSP, 2022.
[47] Feifei Xiong, Stefan Goetze, Birger Kollmeier, and Bernd T Meyer. Joint estimation of reverberation time and carly-to-late reverberation ratio from single-channel speech signals. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 27(2):255 267, 2018.
[48] Cheng Yu, Ryandhimas E. Zezario, Syu-Siang Wang, Jonathan Sherman, Yi-Yen Hsieh, Xugang Lu, Hsin-Min Wang, and Yu Tsao. Speech enhancement based denoising autoencoder with multi-branched encoders. In arxiv, 2020.
13

---

7 Supplementary
In this supplementary material provide the following:
1. A video for qualitative evaluation of model's performance
2. Details regarding AVSpeech-Rooe Rooms curation (7.2 (referenced Sec. 4 of main paper)
3. Details
ablation study with different metric training objectives 773 (referenced in Sec. 5 - "Resllts on AVSpeech-Rooms of main paper)
4. A comparison of audio quality and reverberation metrics between reverberant, de- reverberated, and de-biased audio (7.4)
5. A sample survey slide from our human perception study (Figure 7
6. Model/training details for our RT60 estimator, de-biaser, discriminator, and reverberator
7.5p (referenced Sec. 5 "Implementation Details" of main paper)
7. Details on our data augmentation strategy (7.6) (referenced in Sec. 5 - "Baselines" of main
paper)
8. A brief discussion of our work's 's limitations and broader impact 777 7.8
9. A tabular description of our three-stage training strategy (Table
10. Pseudocode for a discriminator training epoch detailing our reverberator update mechanism (Algorithm []]
11. A visualization illustrating the effect of speaker distance on the Direct-to- Energy Ratio (DRR) in SoundSpaces- Speech audio samples (Figure 9
12. A visualization of the effect of speaker distance on STFT Error in SoundSpaces-Speech
audio samples (Figure 10)
13. A visualization comparing SRMR scores of de-biased and dereverberated data across various reverberation levels for both AVSpeech-Rooe Rooms and SoundSpaces-SSeee (Figure
7.1 Supplementary video
Our video contains several illustrative examples generated by LeMARA on both SuundSpaces-Speeed and AVSpeech- Rooms.. We provide audio generated by the current state-of-the oof-theart art (AViTAR) for reference on each example. We recommend wearing headphones for a better listening experience.
7.2 AVSpeech-Room
Acoustic AVSpeech consists of audio clips from YouTube vidcos along with an RGB image frame selected randomly from the corresponding video clip. To create AVSpeech-Roome we design a of criteria which we use to filter out samples in which the image contains uninformative, non-natural, or misleading acoustic information about the space. We focus on cases in which the room is not visible, a microphone is being used, or virtual background/screen is present any of which will disturb the natural room acoustics for the speaker's voice. We query each sample with our criteria using a Visual Question Answering (VQA) model [20], which we found more reliable than manual annotations we originally obtained on MTurrk contains information about our criteria.
Table 3: Filtering criteria and % of Acoustic AVSpeech samples removed.
dataset
Question
Answer
%
Is a microphone or headset visible in the image?
yes
7.2
Is there a whiteboard/blackboaa in the background?
yes
3.4
Is the entire background one solid color and material?
yes
23.4
Is there large projector screen covering most of the background?
yes
2.2
Is part or all of the background virtual?
yes
1.3
Are there multiple screens in the image? Is the wider room clearly visible?
yes
3.5
no
3.0
14

---

slide, audio clip which best corresponds to the acoustic qualities the audio. The goal identify which space the audio was recorded in. Make disregard the speeker(s) the and only making selection.
image
Human perception study. The instructions given to the user at the start of the survey (left), and sample slide from the survey (right). The user is asked listen the audio clip, and identify which room image most closely matches its acoustics.
Figure 7:
Table 4: Ablation study using different metric training objectives. AR denotes the proposed acoustic residue metric.
LibriSpeech RTE
Metric
SRMR AR
0.2308
0.2156
AR (combined) AR (combined) w/ shortcut
0.2123
0.2100
7.3 Ablations
Table displays our experiments with different self-supervised training objectives. We report performance on the LibriSpeech evaluation setting. The first three rows correspond to experiments which do not utilize the shorcut training strategy (referenced in Sec. 3 "Training" of main paper). Using SRMR alone (row 1) produces the largest (worst) RTE. Training with the acoustic RTE, providing empirical support for
residue metric instead (row leads to a large improvement our metric as an effective training objective. Using our combined metric and the shortcut training strategy (both described in Sec. 3 "Training" of our main paper) further improves the performance by small margin.
7.4 Evaluation of de-biaser performance
evalluats the quality of de-biased audio compared to both reverberant and dereverberated audio. De-biased audio less reverberant than dereverberated audio measured by both absolute RT60 metric and RTE (computed against the ground truth anechoic waveform for SoundSpaces- Speech). De-biased audio also achieves higher SRMR score than dereverberated audio, indicating superior quality and intelligibility.
7.5 Model/training details
RT60 estimator We adopt the RT60 estimator from 3 The estimator takes a spectrogram as input, encodes it with ResNet18 [18), and outputs scalar RT60 estimate. The model is trained on 2.56s clips of reverberant speech simulated on the SoundSpaces platform [4] paired with the ground truth RT60 computed from the RIR used generate the reverberant speech. The model trains using MSE loss between predicted and ground truth RT60 values. Ground truth RT60 is computed using the Schroeder method
39
De-biaser architecture The de- biaser G takes a magnitude spectogram as input. This is passed to bi-directional LSTM with input size 257 and two hidden layers each of size 200, which produces an output with the same temporal length as the input spectrogram. This is passed through linear layer of size 300 and leakyReL.U activation, followed by another linear layer of size 257 and a Sigmoid activation. The final mask is multiplied with the input magnitude spectrogram to create the generated
15

---

Table 5: Evaluation of de-biased audio vs dereverberated audio various metrics.
SoundSpaces-Speett Speech
AVSpeech-Roome
Audio
RT60
RTE
SRMR (t)
RT60 SRMR (1)
Reverberant
0.40
0.36
5.97
0,42
6.76
Dereverberated
0.06
0.02
8.35
0.06
8.99
De- be-biased
0.04
0.01
9.50
0.03
13.14
magnitude spectrogram. A resynthesis module computes phase information from the input audio waveform, combines this with the generated magnitude spectrogram, and performs an inverse STFT to produce the generated waveform. The discriminator D consists of 4 2D Convolutional layers with kernel size (5,5) and 15 output channels, followed by a channel averaging operation and two linear layers of sizes 50 and 10. A LeakyReIUU activation with negative slope 0.3 is used after each intermediate layer. The final layer outputs a scalar-valued metric score estimate.
De-biaser training In stage (1) (see Sec. "Training" of our main paper), we train with batch size 32. During stage (3) fine-tuning, we use batch size of 2. G and D are trained with learning both stages, In each epoch, We train on 10k samples randomly
rates of 2e-6 and 5e-4 respectively selected from the train set without replacement. The reverberator models Rg and Rt are updated with the target networks at a frequency of E = 8 epochs. For all models, we clip each audio sample to 2.56s during training and evaluation.
Reverberator training We train the reverberators with batch size 4 and learning rate of lc-2 in stage (2). During stage (3) fine=tuning, we use batch size and a learning rate of le-6. Both reverberator models and the ViGAS baseline are trained with MSE loss between the log magnitude spectrogram of predicted and ground truth audio.
Baseline training details We use learning rate of 1e-2 and batch size of to train ViGAS. We train AVITAR with batch size 4 - all other hyperparameters are set as described in (3].
Compute All models are trained on 8 NVIDIA Quadro RTX 6000 GPUs
7.6
Augmentation strategy
We follow a data augmentation strategy similar to that proposed in [3] for training the baseline models, which was shown produce better generalization performance on the LibriSpeech setting than when trained without this augmentation strategy. In particular, to each batch of dereverberated audio we add colored noise, perform polarity inversion the waveform with p 0.5, and convolve the waveform with a randomly selected Room Impulse Response (RIR) from a different acoustic environment with p 0.9. At test time, evaluate without these audio augmentations. This strategy designed to mask over residual acoustic information in dereverberated audio during training. We model directly learns remove residual acoustic information, obviating the need for heuristic strategy to mask out.
do this augmentation strategy in approach
7.7 Limitations
Our approach focuses on visual acoustic matching on mono-channel audio exclusively. However, binaural cues in audio play fundamental role in our perception of reverberation and room acoustics We leave it to future work to extend our approach to binaural audio.
g..
7.8 Broader impact
While training on in-the-wild web videos allows wider access diverse variety of speakers and environments, also introduces uncontrolled biases, speaker privacy concerns, and potentially harmful content into the model.
16

---

7.9
Data examples
Refer video to view samples from both BoondSpaces-Speeee and AVSpeech-Roome Rooms.
7.10
LeMARA training procedure
Dataset
Train Step #
SoundSpaces-Specee
AVSpecch-Room
Pre-train G with SRMR,
2
Pre-train R Rp
Pro-train R., Rp
3
Train Ry, R
Train G, Rg, Rs
Figure 8: LeMARA three-step training, Refer to Section 3.4 for detailed description of each step.
7.11
Discriminator epoch - Reverberator update procedure
Algorithm 1 Discriminator training epoch generated data.
R. + R,
Initialize target networks with current reverberator network weights.
R + Rs
4 current epoch #
for ((G(A.) Vi) in generated data botches do {s;} + M([(G(A,), v.)))
Compute metric scores (using R. and Rτ).
D t VLD LLD (D(G(((()))
Update discriminator.
Rt <- Livinna (R)((G((A), v.)),,
Update target networks during disc. training.
R Aisind R((I(G(A), V.)).
end for
if 4%E 0 then
every E epochs, copy target network weights into current reverberators.
R. <- R
Rs R
end
17

---

Direct-fo-Reverberaat Energy Ratio (DRR) vs. Speaker Distance
40
DRR
40
Speaker Distance (m)
Revereerrrer Energy Ratio (DRR) function of the distance between the source and receiver for SoundSpaces-Speee audio samples. As the distance between speaker and receiver increases, the listener receives more reverberant sound than direct sound from the speaker, leading to the observed decrease DRR.
Figure 9: DRR vs. Speaker distance. We plot the
STFT Error vs. Speaker Distance (SoundSpaces-Speech)
AVITAR VIGAS
6.0
LeMARA
4.0
STFT
3.0
2.0
1.0
Speaker Distance (m)
Figure 10: STFT Error vs. Speaker distance. We plot STFT Error as a function of the source- receiver distance for SoundSpaces-Speet samples. LeMARA produces the lowest STFT Error all source-receiver distances.
18

---

Moen SRMR at Varieus Reverberation Levels (SeundSpaces-pooce)
Mean SRMR Various Reverberstion Levels (AVSpoech-Roone Rooms)
de-bie derverb
20.0
12-
Mean SRMR
Meen SRMR
12.5
do-hin
10.0
0.2
0.7
0.8
0.9
1.0
0.2
0.4
1.0
Reverberation Level (RT60)
Reverberation Level (RT60)
Figure 11: De-biaser performance at various reverberation levels. We stratify samples from both datasets by their RT60 and compute the mean SRMR scores of dereverberated and de-biased data within each bin. De-biased audio consistently produces higher SRMR scores than dereverberated audio across all reverberation levels and datasets.
19


Reviews:
[{"review_id": "EAeG39Z48Z", "reviewer": "Reviewer_7iLe", "summary": "This work aims to perform acoustic matching with unpaired training data, i.e. observing the audio only in the target environment without samples of the same audio in the source environment. The basic process involves the common process of dereverberator and reverberator. But one key point in this work is to introduce an acoustic residue metric to measure the residual information in a waveform to ensure the quality of dereverberated audio. Finally, experiments on SoundSpaces-Speech and AV Speech show its effectiveness. ", "questions": "(1) The error of STFT changes obviously in AVSpeech-Rooms compared with other settings, i.e. about 6~7 vs 1~2, what’s the possible reason? I t would be nice to give the clarification. (2) Two questions about the evaluation metrics and the comparison are as listed in weakness.  ", "limitations": "yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presentation is very clear. The motivation to tackle the task is also intuitive and meaningful. The method is reasonable. I like this smart and simple manner proposed in this paper to tackle the key limitation in visual acoustic matching. ", "weaknesses": "\n(1) Is it limited to evaluate with only two metrics, STFT and RTE? (2) Although the number of methods using unpaired data may be few, the number of methods using paired data should be large. There could be some comparison with them to show the performance level of the proposed method. For example, comparing on other paired datasets with existing methods which use paired data could show the difference of the methods use and not use paired data. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the valuable feedback and positive remarks.**\n\n1) **Is it limited to evaluate with only two metrics, STFT and RTE?** \n\nA: We focus on RT60 and Spectrogram loss metrics as these capture well the reverberant acoustic properties of audio, and are used in prior works for acoustic matching [3, 23, 35]. Furthermore, we also provide quantitative results of a human user study for evaluation of perceptual quality in the paragraph starting at line 343.  Based on the reviewers’ helpful suggestions of additional metrics, we now also provide log magnitude STFT error, RT60 bias/correlation coefficient, and relative RT60 error (Tables 5 and 6 in the rebuttal pdf). We also evaluate the de-biaser and dereverberator using a speech quality metric (SRMR) (Table 4 in the rebuttal pdf). These new metrics continue to support our claims, especially on real-world data.\n\n2) **Although the number of methods using unpaired data may be few, the number of methods using paired data should be large. There could be some comparison with them to show the performance level of the proposed method. For example, comparing on other paired datasets with existing methods which use paired data could show the difference of the methods use and not use paired data.**\n\nA: The task of Visual Acoustic Matching (VAM), first introduced in [3], is relatively new, and as such there are very few existing methods, whether using either paired or unpaired data. To the best of our knowledge, we have compared our method against all available relevant works: the original and only Visual Acoustic Matching paper [3], as well as ViGAS [5], a model designed for the related task of novel-view acoustic synthesis that we adapt to for VAM (paragraph at line 294). We also evaluate audio-only variants of these models for comparison against our audio-only model. In addition to these, we evaluate a naive approach in which the dereverberated input is copied to the output. Table 1 displays these baselines in full. If the reviewer has a specific baseline in mind that we have not covered, we would welcome the suggestion and are happy to add it. \n\n3) **The error of STFT changes obviously in AVSpeech-Rooms compared with other settings, i.e. about 67 vs 12, what’s the possible reason? It would be nice to give the clarification.**\n\nA: AVSpeech-Rooms contains a variety of non-speech sounds (e.g. white noise from air conditioning, clicking/tapping noises from object interactions, and even background music) which make proper reverberation even more challenging. These signals may be perceptually weak, but they will show up on a spectrogram and contribute to the larger STFT error which we observe on AVSpeech-Rooms for all methods (Table 1). In contrast, data in SoundSpaces-Speech (derived from a state-of-the-art acoustics simulator) convolves anechoic audio with a Room Impulse Response based on the room geometry (lines 245-252) to produce audio that contains no artifacts or background noises to contribute to spectrogram errors.\n\n\n"}, {"review_id": "NxpBptPiEF", "reviewer": "Reviewer_bz7J", "summary": "This work proposes a method for visual acoustic matching, the task of processing an audio signal so the room acoustics are perceived as originating from within a certain room based on an image. While paired data is generally required for this task, the proposed approach is self-supervised and trained using “in-the-wild” videos of speakers in various rooms. To achieve this, a dereverberation system along with a de-biaser, a visually-conditioned reverberator, and a blind reverberator are used. The de-biaser is trained by optimizing a novel acoustic residue metric that measures the relative difference between the amount of acoustic information in two recordings. While similar to dereverberation, this de-biasing process can be seen as post-processing step that aims to further reduce information in the dereverberated output so the visually conditioned reverberator model must use cues from the supplied image to match the target room acoustics. To use this acoustic residue metric as an objective function, the de-biaser is trained in a MetricGAN fashion where the discriminator is trained to regress the true metric value. Two evaluation datasets are used and results indicate that the proposed method outperforms existing baseline systems. In addition, a human perception study also validates these results.\n", "questions": "Is it fair to call the proposed approach “self-supervised”? While the approach does not require paired data directly, it appears that paired data is indirectly required since pretrained models, such as the RT60 estimator and dereverberation models. In other words, it appears that it would not be possible to train the proposed system in a fully self-supervised manner from scratch, without these pretrained modules, which in fact use labeled/paired data. Instead, it may be more accurate to label the proposed method as one capable of using an “unpaired” dataset instead of self-supervised. Perhaps terms such as “semi-supervised” or “weakly-supervised” are more appropriate? \n\nWhat is the motivation for using the MetricGAN approach to construct a differentiable proxy of the acoustic residue metric $\\mathcal{M}$? Based on the text, it appears that the RT60 estimation model $\\mathcal{RT}$ is implemented with a neural network and hence differentiable. While (3) involves non-differentiable functions such as the absolute value and $\\max$, these are both approximately differentiably, indicating that it may be possible to directly backprop through the metric $\\mathcal{M}$. Is this something the authors considered? If so, was the failure of this approach the motivation to use the MetricGAN approach? Some discussion on the motivation for the MetricGAN approach would be beneficial. \n\n\nLine 30: A reference has not been rendered properly “?”\n", "limitations": "It could be beneficial to include not only the mean RT60 error (RTE (s) as reported in Table 1, but also the bias and correlation coefficient, which is common practice in RT60 estimation evaluation. This can help to establish how the model is failing especially if the test dataset has a non uniform distribution of RT60 targets. \n\nOne limitation of the use of the RT60 estimation in the proposed acoustic residue metric is that this measurement appears to consider only the broadband RT60. It is generally accepted that in most rooms the RT60 can vary quite significantly across frequency, for example, with the higher frequencies decaying at a significantly faster rate than the low frequencies (generally due to absorption from the material properties). As a result, this variation of RT60 across frequency often provides significant insight into the character of the room, which is often detectable by a human listener. Therefore, using only the broadband RT60 in the acoustic residue metric may lead to some cues about the bandwise RT60 remaining in the de-biased recording as long as the broadband RT60 is perturbed. Future work could consider a band-wise RT60 estimation. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "\n1. The proposed method is original and focuses on combining multiple existing approaches along with a novel acoustic residue metric in order to achieve a system that can be trained without paired data. These approaches include the use of various pretrained submodules, including a dereverberation model and a RT60 estimation model, as well as the MetricGAN approach that enables the use of the acoustic residue metric during optimization.\n\n2. As outlined by the authors, existing approaches are often greatly limited by the availability of paired data. Not only is the proposed approach capable of being trained without paired data, the evaluation indicates superior performance to existing methods. \n", "weaknesses": "1. Some relevant references for work in audio-only acoustic matching are omitted. Works such as [1] are similar in nature but rely on conditioning from an audio signal instead of an image. Major components of the reverberator architecture are similar as well, such as the use of a WaveNet architecture for audio processing. This would be highly relevant in the context of the blind reverberator in this work. Also, other audio only approaches such as [2] would be relevant to include in the context of related work. It would also be relevant to address other GAN-based reverberation models such as [3, 4].\n\n2. While overall the manuscript is well organized and clear, further refinement with regards to the explanation of the process for training the complete system may help reduce the burden for readers. For example, perhaps an algorithm listing each step of the process and referencing the variables used in the text would be helpful. \n\n3. While the evaluation considers two datasets and appears relatively sound, it is still somewhat difficult to understand the relative improvement afforded by the proposed method in comparison to the baselines. \n\n4. While the authors are commended for including a human perception study, the design of the study may be somewhat flawed, making it difficult to make any conclusion from the results. \n\n\n[1]\tSu, Jiaqi, Zeyu Jin, and Adam Finkelstein. \"Acoustic matching by embedding impulse responses.\" ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.\n\n[2]\tSteinmetz, Christian J., Vamsi Krishna Ithapu, and Paul Calamia. \"Filtered noise shaping for time domain room impulse response estimation from reverberant speech.\" 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021.\n\n[3]\tRatnarajah, Anton, Zhenyu Tang, and Dinesh Manocha. \"IR-GAN: room impulse response generator for speech augmentation.\" INTERSPEECH (2021).\t\n\n[4]\tRatnarajah, Anton, Zhenyu Tang, and Dinesh Manocha. \"Ts-rir: Translated synthetic room impulse responses for speech augmentation.\" 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the valuable feedback, insights, and positive remarks.**\n\n**Weaknesses:**\n\n1) **Some relevant references for work in audio-only acoustic matching are omitted.**\n \t\nA: Thank you for bringing these audio-only works to our attention.  We are happy to cite them. Our discussion of audio-only acoustic matching is in Lines 30-33, and on Line 79 including NAF ([21]). Our discussion on GAN-based enhancement models is in Lines 94-102 including Sergan ([1]) and MetricGAN models ([11,12,13]).\n\n(Su et al., ICASSP 2020) is relevant audio-only acoustic matching work, which shares our general conditional WaveNet architecture as does [38,32] and the Acoustic Synthesis module of [5] (i.e., it is not unique to (Su et al., ICASSP 2020)). (Su et al., ICASSP 2020) uses training data in which the same utterances are recorded in different environments, motivating an approach that is quite different from our setting, where every (utterance, environment) pair is unique. This key difference motivates the rest of our architectural innovation (adversarial de-biaser, Acoustic Residue metric, dual-WaveNet reverberator, and mutual learning framework).\n\n(Steinmetz et al., IEEE 2021)  addresses the related task of RIR generation along the lines of [35,23]. (Ratnarajah, INTERSPEECH 2021) and (Ratnarajah, IEEE 2021) develop approaches for this same task using conventional image-based GANs. [35,23,11,12,13] collectively address RIR generation and audio-based GAN methods which these two works can be grouped with.\n\n2) **While overall the manuscript is well organized and clear, provide refinement ... perhaps an algorithm listing each step.**\n\nA: Thank you for this suggestion.  We have added an algorithm box in the rebuttal pdf, and a tabular description of our three stage training process. (Figure 14, Table 7).\n\n3) **While the evaluation considers two datasets and appears relatively sound, it is still somewhat difficult to understand the relative improvement afforded by the proposed method in comparison to the baselines.**\n\nA: Table 1 highlights the quantitative improvement in our method over baselines in two datasets using standard metrics (and see Tables 5 and 6 in the rebuttal pdf for additional metrics). We also provided a human user study showing our method outperforms the SOTA (line 343).  Additionally, our demo video (Supp.) contains examples comparing audio generated by our model and the current SOTA (AViTAR [3]) for reviewers to better understand the perceptual improvement.  We believe that this mix of both hard metrics on the one hand and perceptual studies and qualitative examples on the other hand facilitate understanding the relative improvement.  If there is a more specific question from the reviewer, we’d be happy to address it.\n\n4) **While the authors are commended for including a human perception study, the design of the study may be somewhat flawed.**\n\nA: The setup details start at line 343. We believe this is a solid perception study following good practices. However, if we have missed anything, could the reviewer please indicate exactly what might be somewhat flawed?\n\n**Questions:**\n\n1) **Is it fair to call the proposed approach “self-supervised”? Perhaps terms such as “semi-supervised” or “weakly-supervised” are more appropriate?**\n\nA: We appreciate this suggestion and we agree there’s room for different interpretations of the terms.  To explain our thinking: While the RT60 estimator and off-the-shelf dereverberator require supervision for their training, they are treated as frozen modules in our method. As such, our model can be extended to a new dataset with no additional supervision necessary, as we demonstrated with the application to AV-SpeechRooms. Terms such as “semi-supervised” and “weakly-supervised” imply that the amount of paired data required scales with the amount of full training data, whereas our method requires only the finite amount of training data on which the RT60 estimator and dereverberator modules have been trained. However we appreciate the reviewer’s point, and “unpaired” may be a more appropriate term than “self-supervised” given this ambiguity.\n\n2) **Q: What is the motivation for using the MetricGAN approach to construct a differentiable proxy of the acoustic residue metric? Some discussion would be beneficial.**\n\nA: Thanks for the insightful question.  While the Acoustic Residue metric is approximately differentiable, our metric optimizes for both Acoustic Residue as well as SRMR (for both reverberation and speech quality), and the implementation of SRMR we used is not differentiable, motivating the use of the MetricGAN. This balance is important as we found that incorporating SRMR into the metric improves performance beyond a pure Acoustic Residue metric (See ablations table in Supp.), and provides stability during reverberator fine-tuning (lines 232-237). Our approach also allows for extensibility to other non-differentiable speech quality scores such as PESQ.\n\n3) **Q: Line 30: A reference has not been rendered properly “?”**\n\n A: Thank you.\n\n**Limitations:**\n\n1) **It could be beneficial to include not only the mean RT60 error (RTE (s) as reported in Table 1, but also the bias and correlation coefficient.**\n\nA: Thank you for the suggestion. In short, we’ve added the requested error metrics and they reinforce our original claims. See Tables 4, 5 and 6 in the rebuttal pdf.\n\n2) **One limitation of the use of the RT60 estimation in the proposed acoustic residue metric is that this measurement appears to consider only the broadband RT60. Future work could consider a band-wise RT60 estimation.**\n\nA: We thank the reviewer for their insight.  While our results already show consistent gains using the broadband RT60 in the residue metric, we agree it will be interesting future work to explore a band-wise variation. \n\n\n\n"}, {"review_id": "hygsf098bQ", "reviewer": "Reviewer_AZuk", "summary": "This paper addresses the task of \"visual acoustic matching\" (VAM): taking a source audio clip and target visual environment (i.e. an image), and modifying the source audio clip such that it sounds like the clip was recorded in the target environment. The paper proposes a self-supervised approach for training neural networks to solve this task, which can train on examples that only contain the target audio and target visual environment. The idea is to disentangle the room acoustics from the audio content, such that the source room acoustics can be removed, then an audio-visual model can transform this audio to match the target visual environment. The method uses a conditional GAN framework, where an \"acoustic residue\" metric is defined that measures the discrepancy between audio-only and audio-visual reverberation. The acoustic residue metric is used to train a discriminator that can be used to train a \"de-biaser\" generator model that strips additional reverberation from an initially dereverberated signal. The output of the de-biaser can then be fed to the audio-visual reverberator.\n\nAlthough the method can theoretically be applied to other types of signals, the paper focuses on clean speech as the audio signals. Datasets used are SoundSpaces-Speech (reverb impulse responses (RIRs) simulated from Matterport3D scans of homes with 3D-rendered human at source location and audio from LibriSpeech) and AVSpeech-Rooms (YouTube videos that mostly feature a single speaker with little background noise). Objective metrics are used to measure performance: MSE between magnitude spectrograms of predicted and ground-truth speech, and MSE between RT60 estimates of predicted and ground-truth speech. The proposed method achieves better objective scores compared to comparable methods. A perceptual study is also done, which shows users could identify the room from the proposed method 46.1% of the time, versus 34.7% for a baseline method.", "questions": "Q1) Looking at the spectrograms of de-biased audio in Figure 4 and listening in the demo video, de-biased audio seems like it is applying fairly aggressive suppression, perhaps removing some energy of the anechoic audio. This seems like a weakness of the method. Are there mechanisms to prevent over-suppression of the anechoic audio? Does this suggest a trade-off for the de-biaser between dereverberating and suppressing signal? Some discussion of this would be helpful.\n\nQ2) In Figure 2, where does \"A\" come from (lower right panel)? I guess this is the de-biased audio? Would be good to make this more clear", "limitations": "Limitations described adequately.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**S1)** The method provides a means of training self-supervised models for VAM.\n\n**S2)** Evaluation includes a human perceptual study.\n\n**S3)** Demo video is clear and helps with understanding the method. Also, thanks for providing audio demos, they are very useful for evaluating the method and how it compares to the AViTAR baseline.", "weaknesses": "**W1)** The evaluation of the method is weak. Watching the demo video, it seems like this is certainly a difficult task to evaluate, since the effect can be rather subtle. Nevertheless, I think the evaluation metrics could be improved.\n\nFirst, there are better options than just measuring MSE on magnitude spectrograms. Human hearing is logarithmic, so MSE on linear magnitude is a poor match to human perception. An easy alternative is to measure MSE between log spectrograms, although this encounters an issue with where to set the floor on the log to avoid -infinity. A solution to this is to use magnitude raised to a power (e.g. 0.3), which approximates the log, but also goes to 0 where the magnitude is 0. This may help this this issue: \"Models that use this dereverberator without further de-biasing will display artificially low STFT error when evaluated in-dataset\"\n\nSecond, there are other important properties of reverb impulse responses besides RT60, such as direct-to-reverberant ratio (DRR). DRR measures the ratio of energy of the direct path to the energy of the reverberant part, and can be an important cue for distance of the source. Also, distance of the source is not really discussed, and DRR is a crucial property to help measure this (see W3).\n\nThird, the comparison of the RT60 distributions in Figure 4 are not very convincing, because they are just box plots, and it's not clear how the proposed approach \"more closely matches the ground truth target distribution.\": the median is closer to ground-truth than the baseline, but it seems like a different type of plot would allow more detailed comparison of the distributions, like histograms or violin plots.\n\n**W2)** The paper restricts its focus to a clean single speaker, which suggests the method may be limited on real-world data, which can contain a great variety of non-speech sounds, and also multiple speakers. I don't think the proposed method could be applied directly to these scenarios, because all sounds would need to first be separated, then each processed by the pipeline (i.e. each sound dereverberated and re-reverberated)\n\n**W3)** The paper does not discuss the effect of distance of an object, nor the effect of distance on the reverberation. In particular, the DRR is an important property of reverberation that is totally ignored in this paper. At the very least, assumptions about how the method handles distance and location of sources needs to be clearly described and discussed, and also I think evaluation metrics should take distance into account (see W1).\n\n**Minor comments and typos**\n\na) \"data, we\" -> \"data, and we\"\n\nb) \"target space [?,..\": update missing ref\n\nc) \"In-the-wild Web\" -> \"In-the-wild web\"\n\nd) \"We focus on human speech in indoor settings\": clearly specify that it's clean speech, without background noise\n\ne) \"this leaves signals of the target environment\": I think \"residuals\" would be better than \"signals\"\n\nf) \"Unlike SRMR, DNSMOS [31], or any existing off-the-shelf metric that quantifies dereverberation,\": I don't think DNS-MOS is trained to quantify reverberation specifically, it's overall quality, which can be affected by other properties such as background noise and/or artifacts. May be good to adjust this description", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the valuable feedback and questions.  We hope that our clarifications about the dataset contents and incorporation of the additional suggested metrics help reconsider our contribution.**\n\n**Weaknesses:**\n\n**W1)** \n\n**I) There are better options than just measuring MSE on magnitude spectrograms. An easy alternative is to measure MSE between log spectrograms**\n\nA: Thank you for this suggestion.  We have added in log magnitude STFT loss as an additional metric in Tables 5, and 6 in the rebuttal pdf. We again outperform the current SOTA approach (AViTAR [3]) on this metric across both datasets. \n\n**II) There are other important properties of reverb impulse responses besides RT60, such as direct-to-reverberant ratio (DRR). …does not discuss the effect of distance…**\n\nWe discuss the speaker-position dependent nature of RIRs and how this is addressed in SoundSpaces-Speech data creation at Lines 247-255.\n\nDRR estimation from reverberant audio is challenging. Spectrograms mix both direct and reverberant sound components across frequency and time, which makes it difficult to estimate DRR even if pre-training the model on synthetic audio. Spectrogram-based RT60 estimation is more reliable, as it can be accurately estimated from energy reduction across temporal bins on the spectrogram.  If we have misunderstood the suggestion, we’d welcome a pointer on how to allow DRR as a reliable metric when one lacks ground truth RIRs.\n\nTo illustrate the effect of speaker distance on performance, we provide an STFT Error vs speaker distance plot, Figure 16 in the rebuttal pdf (for SoundSpaces, where we have GT RIRs). Overall, we observe minimal change in error as a function of speaker distance. To provide an analysis of DRR on our data, Figure 15 displays the relationship between speaker-distance and DRR on samples from SoundSpaces-Speech. We observe that DRR generally decreases as a function of speaker distance.\n\n**III) RT60 distributions in Figure 4 are not very convincing, because they are just box plots…a different type of plot would allow more detailed comparison of the distributions, like histograms or violin plots.**\n\nA: Thank you for the valuable suggestion for how to best present the output distributions.  We now provide histogram and violin plots of the source, predicted, and target RT60 distributions (Figures 9-12 in rebuttal pdf).  We believe this is indeed a more direct way to see that the proposed approach more closely matches the target distribution on all three setups, compared to our original boxplots.\n\n**W2)**\n\n **The paper restricts focus to a clean single speaker, which suggests the method may be limited on real-world data, which can contain a great variety of non-speech sounds, and also multiple speakers.**\n\n A: Many real-world videos consist of a single speaker (instructional videos, presentations, vlogs).The AVSpeech dataset from which our training set is sampled consists of over 290k such videos from YouTube. Although these videos are single speaker, we disagree with the characterization as little background noise.  The clips contain a variety of non-speech background sounds (e.g. white noise from air conditioning, clicking/tapping noises from object interactions, background music).  Our model performs well in this real-world setting, as shown by the results on AVSpeech-Rooms in Tables 1 (main paper), 4,5, and 6 (rebuttal pdf) in which we consistently outperform the SOTA. \n\nConceptually our approach does not make any assumptions about the type of audio—whether speech or single speaker (see Line 67, footnote 2). Given a dereverberator and RT60 estimator trained on the relevant types of sounds (mixtures of speakers and/or non-speech sounds), our approach can be used to train a VAM model directly on these mixtures. In principle, audio source separation for sources close by in the same environment would not be necessary; they would all be influenced by the same room acoustics.  We leave such explorations for future work.\n\n**W3)**\n\n**The paper does not discuss the effect of distance of an object, nor the effect of distance on the reverberation...(see W1)**\n\nA: Please refer to our earlier response in **W1 part II**.\n\n**Minor comments and typos**\n\nA: Thank you, fixed.\n\n\n**Questions:**\n\n**Q1) Are there mechanisms to prevent over-suppression of the anechoic audio? Does this suggest a trade-off for the de-biaser between dereverberating and suppressing signal? Some discussion would be helpful.**\n\nA: Thank you for the insightful questions. The de-biaser may remove some energy from audio, but the spectrogram-based loss provides a strong signal for the reverberator to add this energy back during fine-tuning. This can be seen in the predicted reverberant audio spectrograms in the demo video. There does exist a trade off between dereverberation and suppression: if the de-biaser's aggressive dereverberation also removes energy from audio, then the reverberator fine-tuned on this data may learn to add more energy back than is necessary, which at test time may hurt STFT error. Optimizing de-biased speech for SRMR helps mitigate this, preserving low frequency modulation energy - speech content - over high frequency modulation energy (reverberant content). Pre-training with SRMR provides the generator with a strong prior for preserving energy associated with speech content in de-biased audio, guarding against indiscriminate energy suppression during Acoustic Residue fine-tuning. Also, energy removal may contribute to larger STFT based error, but it does not affect reverberant properties, which our human user study shows may be more important in evaluating perceptual accuracy (Line 343).\n\n**Q2) In Figure 2, where does \"A\" come from (lower right panel)? I guess this is the de-biased audio? Would be good to make this more clear.**\n\nA: “A” is the input to the metric, which can be either the de-biased audio or the reverberant audio. We have updated the figure to more clearly show this.\n\n\n\n\n\n\n\n\n"}, {"review_id": "3wtaVDlny2", "reviewer": "Reviewer_Ky2n", "summary": "The paper introduces a self-supervised method for visual acoustic matching (VAM), where the training samples consist of only the target scene image and audio. The paper is well-written and easy to follow. The experimental results have validated the effectiveness of the proposed approach. We have the following comments: \n(1) The proposed system is somewhat simple as it incorporates existing methods. Therefore, the scientific depth presented in the paper may not be up to the standards presented at the prestigious conference NeurIPS.\n(2) It would be beneficial to include an analysis of the dereverberator. The authors can evaluate the dereverberator's performance using SRMR with different levels of reverberation. It would also be interesting to see the results with and without the De-biaser component based on SRMR. \n(3) Please provide results for environments with mild, moderate, and severe reverberant conditions. This will help demonstrate the effectiveness of the LeMARA approach more clearly. \n(4) It is crucial to showcase the impact of the De-biaser component. In addition to the spectrograms presented in Figure 4, I recommend that the authors perform additional experiments using supplementary metrics like SRMR or a readily available ASR model. This will provide further evidence and insight into the effectiveness of the De-biaser approach.\n(5) It would be informative to report the results of the LeMARA approach separately for seen and unseen images. \n(6) Figure 4 shows that LeMARA exhibits greater variation in RT60 compared to AViTAR. Please provide an explanation for this observation. \n(7) We suggest preparing an anonymous website where several sets of sound samples of the source, target, AViTAR, and LeMARA can be presented. \n(8) Figure 5 indicates that AViTAR outperforms LeMARA in the left-down example. Please provide an explanation for this result.", "questions": "(1) A critical point is to showcase the effectiveness of the De-biaser component. Please provide additional experiments to demonstrate its performance.  \n(2) It s interesting to see the achievable performance of LeMARA approach under mild, moderate, and severe reverberant conditions. \n(3) It is crucial to report the results of the LeMARA approach separately for seen and unseen images.", "limitations": "(1) The proposed system is somewhat simple as it incorporates existing methods. Therefore, the scientific depth presented in the paper may not be up to the standards presented at the prestigious conference NeurIPS.\n(2) Figure 4 shows that LeMARA exhibits greater variation in RT60 compared to AViTAR. Please provide an explanation for this observation. \n(3) Figure 5 indicates that AViTAR outperforms LeMARA in the left-down example. Please provide an explanation for this result.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a self-supervised method for visual acoustic matching (VAM), where the training samples consist of only the target scene image and audio. The paper is well-written and easy to follow. The relevance of the research task holds significant value in AR/VR systems and can serve as a crucial element. The experimental results have validated the effectiveness of the proposed approach.", "weaknesses": "Some analyses, which can further demonstrate the effectiveness of the proposed approach, are missing, for example:\n(1) The authors can evaluate the dereverberator's performance using SRMR with different levels of reverberation. It would also be interesting to see the results with and without the De-biaser component based on SRMR. \n(2) Please provide results for environments with mild, moderate, and severe reverberant conditions. This will help demonstrate the effectiveness of the LeMARA approach more clearly. \n(3) It is crucial to showcase the impact of the De-biaser component. In addition to the spectrograms presented in Figure 4, I recommend that the authors perform additional experiments using supplementary metrics like SRMR or a readily available ASR model. This will provide further evidence and insight into the effectiveness of the De-biaser approach.\n(4) It would be informative to report the results of the LeMARA approach separately for seen and unseen images. \n(5) In Line 30, there is one reference missing.\n(6) In Fig. 2(b), D(G(A_T)) should be D(G(A_t)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the valuable feedback and positive remarks.**\n\n**Weaknesses:**\n\n**1) The authors can evaluate the dereverberator's performance using SRMR with different levels of reverberation. It would also be interesting to see the results with and without the De-biaser component based on SRMR.**\n\nA: Thanks for the suggestion.  This is easy to add to our analysis.  Figure 7 and Table 4 in the rebuttal pdf shows the SRMR score of dereverberated and de-biased data at various levels of reverberation. Overall, the de-biaser produces audio that is significantly cleaner and more anechoic than de-reverberated data.\n\n**2) Please provide results for environments with mild, moderate, and severe reverberant conditions.**\n \nA: Figure 8 in the rebuttal pdf shows Relative RT60 Error across “bins'' of increasing reverberation. Relative RTE generally increases with reverberation across both datasets, likely because highly reverberant audio contains strong residual acoustic clues that may be difficult to completely remove, whereas this is easier in audio that has less reverberation to begin with. On AVSpeech-Rooms and LibriSpeech generalization, LeMARA achieves lower RTE than the baselines even in this high reverberation regime, due to the de-biaser’s ability to strip away even strong residual acoustic clues in audio.\n\n**3) It is crucial to showcase the impact of the De-biaser component.**\n \nA: Our results focus on the correctness of the reverberator outputs (Table 1 in the main paper, Tables 5 and 6 in rebuttal pdf), given that the task is visual acoustic matching, i.e., re-synthesizing reverberation for a new environment.  In particular, rows 3 and 5 in Table 1 of the main paper pinpoint the impact of the de-biaser.  Furthermore, our ablations table in Supp. confirms the effectiveness of our novel de-biaser.  However, we appreciate that the reviewer is interested in understanding how good the “under the hood” de-biaser module is itself.  To that end, Table 4 in the rebuttal pdf highlights the impact of de-biasing on speech quality and reverberation across a variety of metrics, including SRMR. Figure 13 shows the distribution of SRMR scores for reverberant, dereverberated, and de-biased audio. Especially on real-world data (AVSpeech-Rooms), we observe that de-biased data has significantly better quality than dereverberated data. \n\n**4) Informative to report the results of the LeMARA approach separately for seen and unseen images.**\n\nA: Thanks for raising this point. We report results on seen data in Table 6 in the rebuttal pdf. Our LeMARA method outperforms baselines on a variety of RT60 and STFT based metrics across both datasets. Please refer to Table 1 (main paper) and Table 5 (in rebuttal pdf) as well Lines 305-327 for the results and original analysis on unseen data.We reported test-unseen in the main paper as this is the more robust evaluation of performance and due to space constraints.\n\n**5) In Line 30, there is one reference missing.** \n**6)In Fig. 2(b), D(G(A_T)) should be D(G(A_t)).**\n\nThank you, fixed.\n\n**Questions:**\n\n**1) Critical to showcase the impact of the De-biaser component.**\n\nA: Please see our response for **Weaknesses Question 3**.\n\n**2) Interesting to see the achievable performance of LeMARA approach under mild, moderate, and severe reverberant conditions.**\n\nA: Please see our response for **Weaknesses Question 2**.\n\n**3) Crucial to report the results of the LeMARA approach separately for seen and unseen images.**\n\nA: Please see our response for **Weaknesses Question 4**.\n\n**Limitations:**\n\n**1) Proposed system is somewhat simple as it incorporates existing methods.** \n\nA: Existing approaches for VAM [3] use an off-the-shelf model dereverberator to pre-process speech before acoustic matching. In contrast, we propose a mutual-learning based approach that cyclically optimizes a de-biaser to strip away acoustics and a reverberator to add these acoustics back in. We construct a time-domain GAN model and novel adversarial training objective (Acoustic Residue) which assigns a scalar value to the amount of residual acoustic information in an audio clip. We build a dual-WaveNet model to represent the Acoustic Residue function (see Figure 2c), consisting of a blind WaveNet model and a visual-conditioned WaveNet with RT60 estimation network heads, which are used to compute the Acoustic Residue loss signal. Finally, we also devise a novel training strategy for our GAN (Line 209) that uses the Acoustic Residue signal to not only optimize the generator component as in a traditional GAN, but to also update the Acoustic Residue networks themselves, mitigating the distribution shift that occurs in generated data over the course of GAN training. Please refer to Lines 158, 209, 130, and 120 for details on these novel elements.\n\n**2) Figure 4 shows that LeMARA exhibits greater variation in RT60 compared to AViTAR. Please provide an explanation.**\n\nA: This is an important attribute of our approach.  Both AViTAR [3] and ViGAS [5] perform VAM on dereverberated audio that has residual acoustic reverberation. Thus they learn to add in less reverberation than would be necessary if training on true anechoic data. At test time, however, this leads to under-reverberation, as seen prominently in Figure 11 in the rebuttal pdf.  In contrast, our LeMARA is trained on de-biased audio that has been adversarially optimized to strip away reverberation (Lines 56-59). Given this pseudo-anechoic audio and the natural variation of RT60 values in our training data, the reverberator in LeMARA correctly learns to add a wider variation of reverberation levels into the audio, conditioned on the image.\n\n**3) Figure 5 indicates that AViTAR outperforms LeMARA in the left-down example. Please provide an explanation.**\n\nA: We speculate that the irregular room shape along with camera lens distortion makes the room appear artificially small, leading the model to under-reverberate the audio (Lines 340-342).\n\n\n\n\n\n\n"}, {"review_id": "bwBgrLCJo3", "reviewer": "Reviewer_2YZ1", "summary": "The authors proposed a self-supervised approach to match acoustic conditions via visual information without the need of paired audio-visual data. With this approach, the in-the-wild web data or simulated data can be utilized. They show that this approach can outperforms the state-of-the-art on multiple datasets.", "questions": "- The use of RT60 needs more explanation, is the intuition to model just the acoustic environment and disentangled with the content involved?\n- Why is Table 1 on the far right, train with AVSpeech-Rooms and test with LibriSpeech does not include comparisons of STFT?", "limitations": "- No potential social or ethical implications.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed approach is novel in several aspects, 1. does not require paired audio-visual data, 2. Able to leverage more data from the web or from simulation.\n- The authors provide human perception study, which is important for their claims in perception accuracies.\n- This technique has potential applications in various audio domains such as domain adaptation, help improving in-the-wild acoustic event classification, etc.", "weaknesses": "- The writing of this paper is not easy to follow and missing content, for example in line 200 it refers to Section 4 for details in \"RT60 to allow generalization\", which does not existing in Section 4, mostly focusing on datasets. In line 226, the off-the-self dereverberator is referred to Section 5, which is difficult to locate for the readers.\n- In Section 5, for all the implementation details, suggest the authors to organize and highlight with a table, what data is used to train in which step. Currently it is not easy to follow in the scripts and have clear understanding of the experimental designs.\n- The paragraph of line 188 seems to appear in the wrong place, referred Figure 4 is in the results section, while the narrative appears in the approach section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the valuable feedback.**\n\n**Weaknesses:**\n\n**1) The writing of this paper is not easy to follow and missing content, for example in line 200 it refers to Section 4 for details in \"RT60 to allow generalization\", which does not existing in Section 4, mostly focusing on datasets. In line 226, the off-the-self dereverberator is referred to Section 5, which is difficult to locate for the readers.**\n\nA: Thank you for pointing out the error in line 200; this has been corrected. Line 226 is on page 6, Section 5 (“Experiments”) can be found on the next page. The second paragraph describes the off-the-shelf dereverberator. Clicking on the section reference also directs here.  We respectfully point out that the other 4 reviewers rate our Presentation clarity as Good or Excellent.\n\n**2) It is not easy to follow in the scripts and have clear understanding of the experimental designs. In Section 5, for all the implementation details, suggest the authors to organize and highlight with a table, what data is used to train in which step.**\n\nA: We thank the reviewer for their suggestion. We have added Table 7 in the rebuttal pdf which displays the training steps for each dataset in concise form.  \n\n**3) The paragraph of line 188 seems to appear in the wrong place, referred Figure 4 is in the results section, while the narrative appears in the approach section.**\n\nA: We placed this paragraph with references to Figure 4 immediately after introducing the concept of de-biasing in order to help the reader intuitively grasp what the de-biaser is doing to audio visually, before they progress with the rest of the method.\n\n**Questions:**\n\n**1) The use of RT60 needs more explanation, is the intuition to model just the acoustic environment and disentangled with the content involved?**\n\nA: We describe our reasoning for using RT60 in the paragraph at line 194. RT60 is a content-invariant measure of room reverberation, and is a function of room geometry and surface material absorption and reflection properties that characterize a room. These features make RT60 the optimal metric for our task, where we are evaluating whether two waveforms with potentially mismatched content audio sound as if they were recorded in the room.  The metric is used frequently in the literature, and particularly for evaluating acoustic matching quality in prior work [35,23,19].\n\n**2) Why is Table 1 on the far right, train with AVSpeech-Rooms and test with LibriSpeech does not include comparisons of STFT?**\n\nA: STFT is not applicable for that experiment, as explained in Line 291-293.  The LibriSpeech experiment (Table 1, last two columns) performs visual acoustic matching using anechoic source speech samples from the LibriSpeech dataset and real-world target images from AVSpeech-Rooms. Thus the predicted and ground truth reverberant output in this experiment have different speech content, so the STFT metric is not applicable.  Applied here, STFT would capture differences in audio *content* (which by definition must be different here) instead of measuring error in acoustic properties alone. Hence, for this setting we report errors in the reverberant properties of the audio (RT60 error statistics), as well perceptual accuracy via the human user study.\n\n\n\n"}]

Author rebuttal:
Thanks to all reviewers for their time and valuable feedback.  

Three reviewers recommend accepting.  The other two suggest additional error metrics of interest and ideas for how we plot the results (Reviewer AZuk) and easy-to-address clarifications and items addressed already in the text (Reviewer 2YZ1).  We address all items below and show the requested metrics in the rebuttal pdf.

Meta-review: All reviewers recommend acceptance. The AC sees no basis to overturn the reviews, and thus recommends acceptance. Authors should attend to main points in the reviews, such as discussing suppression (Reviewer AZuk) and discussion of the evaluation metrics (Reviewer 2YZ1), when preparing a final version.
Decision: Accept (poster)