Learning Audio-Visual Dereverberation
Authors: Changan Chen
Wei Sun
David Harwath
Kristen Grauman
Venue: ICLR 2022 Submitted (2022)
Keywords: speech enhancement
audio-visual learning
speech dereverberation
room acoustics

Abstract: Reverberation from audio reflecting off surfaces and objects in the environment not only degrades the quality of speech for human perception, but also severely impacts the accuracy of automatic speech recognition. Prior work attempts to remove reverberation based on the audio modality only. Our idea is to learn to dereverberate speech from audio-visual observations. The visual environment surrounding a human speaker reveals important cues about the room geometry, materials, and speaker location, all of which influence the precise reverberation effects in the audio stream. We introduce Visually-Informed Dereverberation of Audio (VIDA), an end-to-end approach that learns to remove reverberation based on both the observed sounds and visual scene. In support of this new task, we develop a large-scale dataset that uses realistic acoustic renderings of speech in real-world 3D scans of homes offering a variety of room acoustics. Demonstrating our approach on both simulated and real imagery for speech enhancement, speech recognition, and speaker identification, we show it achieves state-of-the-art performance and substantially improves over traditional audio-only methods.

Under review as a conference paper at ICLR 2022
LEARNING AUDIO- VISUAL DEREVERBERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Reverberation from audio reflecting off surfaces and objects in the environment not only degrades the quality of speech for human perception, but also severely impacts the accuracy of automatic speech recognition. Prior work attempts to remove reverberation based on the audio modality only. Our idea is learn dereverberate speech from audio-visual observations. The visual environment surrounding human speaker reveals important cues about the room geometry, materials, and speaker location, all of which influence the precise reverberation effects in the audio stream. We introduce Visually-Infone Informed Dereverberation Audio (VIDA), an end-to-end approach that learns remove reverberation based on both the observed sounds and visual scene. In support of this new task, we develop a large-scale dataset that uses realistic acoustic renderings of speech in real-world 3D scans of homes offering a variety of room acoustics. Demonstrating our approach both simulated and real imagery for speech enhancement, speech recognition, and speaker identification, we show it achieves state-of-the-arr performance and substantially improves over traditional audio-only methods.
INTRODUCTION
Audio reverberation occurs when multiple reflections from surfaces and objects in the environment build up then decay, altering the original audio signal. While reverberation bestows a realistic sense of spatial context, it also can degrade a listener's experience. In particular, the quality of human speech greatly affected by reverberant environments as illustrated by how difficult it can be to parse the words of family member speaking loudly from another room in the house, tour guide describing the artwork down the hall of magnificent cavernous cathedral, or a colleague participating in a Zoom call from a cafe. Similarly, automatic speech recognition (ASR) suffers when given reverberant speech input (Kinoshita et al.) 2016, Szöke et al., 2019, Watanabe al.) 2020). Thus there is great need for dereverberation algorithms that can strip away reverb effects for speech enhancement, recognition, and other downstream tasks, which could in turn benefit many applications in teleconferencing. assistive hearing devices, augmented reality, and video indexing.
The audio community has made steady progress on speech dereverberation (Han et al. 2015 Wu et al. 2016 Zhao al. (019b). While past approaches have tackled the problem with signal processing and statistical techniques (Nakatani et al. 2010 Naylor & Gaubitch| 2010), many modern approaches use neural networks to learn mapping from reverberant clean spectrograms (Han al. 2015] Ernst et al. 2018; Fu et al.) 2019). To our knowledge, all existing models for dereverberation rely purely on audio. This underconstrains the dereverberation task since the latent parameters of the recording space are not discernible from the audio alone.
However, we observe that in many practical settings of interest-video conferencing, augmented aadexing-reverberann reverberant audio is naturally accompanied by visual (video) stream. Importantly, the visual stream offers valuable cues about the room acoustics affecting reverberation: where are the walls, how are they shaped, where is the human speaker, what is the layout of major furniture, what are the room's 's dominant materials (which affect absorption, reflection, and refraction), and even what is the facial appearance and/or body shape of the person speaking (since body shape determines the acoustic propertics of person's 's speech, and reverb time is frequency dependent). For example, reverb is typically stronger when the speaker is further away; speech is more reverberant in a large church or hallway; heavy carpet absorbs more sound. See Figure
reality, Web video indexing
Our idea is to learn to dereverberate speech from audio-visude observations. In this task, the input reverberant speech and visual observations of the environment surrounding the human speaker, and the output is a prediction of the clean source audio. To tackle this problem, there are two key

---

Under review as a conference paper at ICLR 2022
Anechoic Chamber
Church, close speaker
Classroom, close speaker
Classroom, distant speaker
Visual cues reveal key factors influencing reverb effects on human speech audio. For example, these audio speech samples (depicted as waveforms and spectrograms) are identical lexically, but have very different reverberation properties owing to their differing environments. In the church, reverb is strong, in the classroom it is less, and when the speaker is distant from the camera it is again more evident. We propose audio audio-visual dereverberation to learn from both modalities how strip away reverb effects, therehy enhancing speech quality, recognition, and speaker identification.
Figure I:
technical challenges. First, how model the multi-modal dereverberation process in order infer the latent clean audio. Second, how to secure appropriate training data spanning variety of physical environments for which we can sample speech with known ground truth mnonrreverberant, anechoic) audio. The latter is non-trivial because ordinary audio/video recordings are themselves corrupted by reverberation but lack the ground truth source signal we wish to recover.
For the modeling challenge, we introduce an end-to-end approach called Visually-Informn Dere- verberation of Audio (VIDA). VIDA consists of Visual Acousties Network (VAN) that learns reverberation properties of the room geometry, object locations, and speaker position. Coupled with a multi-modal UNet dereverberation module, learns to remove the reverberations from single-channel audio stream. In addition, we propose an audio-visual (AV) matching loss to enforce consistency between the visually-inferred reverberation features and those inferred from the audio signal. We leverage the outputs of our model for multiple downstream tasks: speech enhancement, speech recognition, and speaker verification.
Next, to address the training data challenge, SoundSpaces (Chen et al. 2020), a 3D simulator for real-world scanned environments that al- lows both visual and acoustic rendering. We insert "clean" audio voices together with 3D humanoid model at various positions within an array of indoor environments, then sample the images and properly reverberating audio when placing the receiver mierophone and camera at other positions in the same house. This strategy allows sampling realistic audio-visual instances coupled with ground truth raw audio train our model, and it has the added benefit of allowing controlled studies that vary the parameters of the capture setting. As we will show, the data also supports sim2real transfer.
develop new large-scale dataset using
We first train and evaluate our model on 82 large-scale scale environments each multi-roue room home con- taining a variety of objects-coupled with speech samples from the LibriSpeech dataset (Panayotoy al. 2015), then further test with real-world data. We consider both near-field and far-field settings where the human speaker is in-view or quite far from the camera, respectively. The proposed model outperforms methods restricted to the audio stream, improves the state of the art for multiple tasks with speech enhancement, and transfers successfully to real-world data.
Our main contributions are to I) present the task of audio-visual dereverberation, 2) address it with a new multi-modal modeling approach and novel reverb-visuse matching loss, 3) provide benchmark evaluation framework built on both SoundSpaces and real data, and 4) demonstrate the utility of AV dereverberation for multiple practical tasks. Overall, our work shows the potential for speech enhancement models to leverage room acoustics by seeing the 3D environment.
2 RELATED WORK
Audio dereverberation and speech enhancement. Audio dereverberation and speech enhance- ment have long and rich literature (Neely & Allen 1979. Miyoshi & Kaneda 1988] Naylor & Gaubitch 2010 Kinoshita et al. 2016). While dereverberation can be done with microphone arrays, focus single audio channel approaches, which require fewer assumptions about the input
2

---

Under review as a conference paper at ICLR 2022
data. Recent deep learning methods achieve promising results to dereverberate (Han et al.
2015
Wu et al., 2016; Zhao et | 2019b] Su et al. 2020b), denoise (Xu et al.) 2019: Fu et al. 20 9 Su
et a 2020b), or separate (Hershey et al., 2016 Sttller et al. 2018) the audio stream using audio
input alone, which can improve downstream speech recognition Ko et al. 2017; K Kinoshita et al.
2016 Watanabe et al.) (020) and speaker recognition (Snyder et al. 2018). Acoustic simulations
can provide data augmentation during training (Kinoshita et al. 2016 Ko et al., 2017) Zhao et al. 2020). Some work targets "room-aware" deep audio features capturing reverberation properties (e.g..
RT60) Giri et al. (015),, or synthesizes reverberation effects via acoustic matching (Su et al., 2020a).
To our knowledge, the only prior work drawing on the visual stream to infer dereverberated audio limited to using lip regions on near-field faces to first separate out distractor sounds (Tan et al., 2020),
and does not model anything about the visual scene for dereverberation purposes. In contrast, our model accounts for the full visual scene, far-field speech sources, and even out-of-view speakers. Our approach is the first to learn visual room acoustics for dereverberation, and it yields statc-of-the- results with direct benefits for multiple downstream tasks.
Visual understanding of room acoustics. The room impulse response (RIR) is the transfer func- tion capturing the room acoustics for arbitrary source stimuli; once convolved with a sound waveform, it produces the sound of that source in the context of the particular physical space. While traditionally measured with specialized equipment in the room itself (Stan et al. 2002 Holters et al. 2009) or else simulated with sound propagation models (Allen & Berkley 1979 Chen et al. 2020; Murphy et al. 2007), recent work explores estimating an RIR from an input image using CNNs (Kon & Koike 2019) conditional GANs (Singh et al.) 2021) in order to simulate reverberant sound for given environment. Video-based methods can lift monaural audio into its spatialized (binaural, ambisonic) counterpart in order to create an immersive audio experience for human listeners (Morgado et al.)
2018
Gao & Grauman 2019 Li et al. 2018).
Such methods share our interest in learning the visual properties of scene that influence the audio channel. However, unlike any of the above methods, rather than generate spatialized audio to benefit human listeners in augmented or virtual reality, our goal to dereverberate audio- removing the effects of the room acoustics-to to benefit automatic speech analysis. In addition, prior methods use imagery taken at camera positions at an unknown offset from the microphone, i.e., conflating all RIRs for scene with one image, which limits them to coarse characterization of the environment (Marco Jeub 2009 Jeub et al. 2010; Murphy Shelley 2010). In contrast, our data and model align the camera and microphone to capture novel fine- -oee-gganndd audio- properties, including the human speaker's 's location with respect to the microphone when the speaker is in view.
Audio-visual embodied learning in realistic simulations. Recent work in embodied AI explores how vision and sound together can help agents move intelligently in 3D environments. Driven in part by new tools for audio-visual (AV) simulations in realistic scanned environments (Chen et al., 2020), new research develops deep reinforeement learning approaches to train agents to navigate to sounding objects (Chen et al.] 2020) Gan et al.) (020), explore unmapped environments (Dean et al. 2020), or move around to better separate multiple overlapping sounds in a house (Majumder et al. 2021). Our work also leverages state-of tate-offthhe-rr the-art AV simulations for learning, but our objective and models are entirely different. Rather than train virtual robots to move intelligently, our aim to clean reverberant audio for better speech analysis.
Audio-visual learning from video. Multi-modal video understanding has experienced a resurgence of work in the vision, audio, and machine learning literature in recent years. This includes exciting
advances in self-supervised cross-modal feature learning from video (Morgado et al.) 2020 A Awaassll
ct al.
2020 Korbar et al., 2018). localizing objects in video with both sight and sound (Hu et al.
2020). and audio-visual speech enhancement or separation (Ephrat et al. 20 8 Owens& & Efros 2018
Zhao et al. 2019a| Afouras et al., 2018 2020, Michelsanti et al.) 2020 Sadeghi al. 2020
Zhou
et al. 2019
Hou et al. 2017). None of these methods address speech deverberation.
3
THE VISUAL. DEREVERBERATION TASK
We introduce the novel task of audio-visual dereverberation. In this task, speaker (or other sound source) and a listener are situated in 3D environment, such as the interior of a house. The speaker- whose location is unknown to the listener ppronccessssseee speech waveform Aş A superposition of the direct sound and the reverb is captured by the listener, denoted A,. The reverberant speech A, can be modeled as the convolution of the anechoic source waveform A, with the room impulse response
3

---

Under review as a conference paper at ICLR 2022
Panorama
Spectrogram
Camera
Speaker
Figure 2:
audio-visual rendering for a Matterport environment. Left: bird's s-eye view of the 3D environment.
Right: panorama image rendered at the camera location and the corresponding received speetrogram.
(RIR) R, i.. (t) (t) R(t) (Neely & Allen 1979). R is a function of the environment's geometry, its materials, and the positioning of the speaker and the listener. It is possible in principle to measure the RIR R for real-world environment, but doing so can impractical when the source and listener are able to move around or must cope with different environments. Furthermore, in the common scenario where we want to process video captured in environments to which we have no physical access, measuring the RIR is simply impossible.
Crucial to our task, we consider an alternative source of information about the environment: vision. We assume the listener observes its surroundings from an RGB-D camera an RGB camera coupled with single-image depth estimation (Eigen et al. 2014 Godard et al. 2019). Intuitively, we should be able to leverage the information about the environment's geometry and material composition that is implicit in the visual stream- as well as the location of the speaker (if visible) to estimate its reverberant characteristics. We anticipate that these cues can inform an estimate of the room acoustics, and thus the clean source waveform. Given the RGB I, and depth image Ia captured by the listener from its current vantage point, the task is predict the source waveform A, from the images and reverberant audio: (t) = fpT, Ia, (t)). This setting represents common previously discussed, and poses new challenges for speech enhancement and recognition.
scenarios
4 DATASET CURATION
For the proposed task, obtaining the right training data is itself a challenge. Existing video data contains reverberant audio but lacks the ground truth anechoic audio signal, and existing RIR datasets (Marco Jeub, [0099] Jeub al., 2010 Murphy & Shelley] 2010) do not have images paired with the microphone position. We introduce both real and simulated datasets to enable reproducible research on audio-visus visual deverberation.
3D environments and audio simulator. First we introduce a large-scale dataset in which we couple real-world visual environments with state-of-the-arr audio simulations accurately capturing the environments' spatial effects on real samples of recorded speech. We want our dataset to allow control of a variety of physical environments, the positions of the listener/camera and sources, and the speech content of the sources-as all while maintaining both the observed reverberant A, (t) and ground truth anechoic (t) sounds. To this end, we leverage the audio-visual simulator SoundSpaces (Chen et al. 2020), which provides precomputed RIRs R(t) on uniform grid of resolution m for the real-world environment scans in Replica (Straub et al. 2019) and Matterport3D (Chang et al. 2017). Material modeling is realized by mapping object classes to their reflection and absorption coefficients. We use 82 Matterport environments due to their greater scale and complexity; each environment has multiple rooms spanning on average 517 m The rooms' visual and geometric variety offers an opportunity to learn natural acoustic associations (see distribution of reverberation times in Supp.).
Reverberant speech in 3D visual environments. We extend SoundSpaces to construct reverberant speech. As the source speech corpus we use LibriSpeech (Panayotov et al.) (015)), which contains 1,000 hours of 16kHz English speech from audio books. We train models with the train-clcan-360 split, and use the dev-clean and test-clean sets for validation and test splits, respectively. Note that these splits have non-overlapping speaker identities. Similarly, we the standard disjoint vrant/valtesst splits for the Matterport 3D visual environments (Chen et al.] (022). Thus, neither the houses nor speaker voices observed at test time are ever observed during training.
For each source utterance, we randomly sample a source-receiver location pair in a random environ- ment, then convolye the speech waveform A, (t) with the associated SoundSpaces RIR R(t) obtain
4

---

Under review as a conference paper at ICLR 2022
ResNert8
Visual Acoustics Network
1X Cony
RGB Image
Reverb visuel
Matching loss
ResNorte
Magnitude & phase loss
STFT
Depth Image
ISTFT
Segment
Siitch
Reverberant Spedtrogram
Spectrogram
Dereverberation
Figure 3: VIDA model architecture. We convert the input speech to a spectrogram and use overlapping sliding windows obtain 2.56 second segments. For visual inputs, we separate ResNet18 networks to extract features e" and e", which are fused to obtain e We feed the spectrogram segment S to UNet encoder, tile and concatenate e with the encoder's output, then use the UNet decoder to predict the clean spectrogram St. During inference, we stitch the predicted spectrogams into full spectrogram and use Griffin-Lim (Griffin & Lim 1984) to reconstruct the dereverberated waveform.
the reverberant (t). To augment the visual stream, we insert 3D humanoid of the same gender as the real speaker at the speaker location and render RGB and depth images at the listener location. We consider two types of image: panorama and normal field of view (FoV). For the panorama image, we stitch 18 images each having horizontal FoV of 20 degrees, for a full image resolution of 192 756. For the normal FoV, we render images with a 80 degree FoV, at a resolution 384 256. While the panorama gives fuller view of the environment and thus should allow the model better estimate the room acoustics, the normal FoV is more common existing video and thus will facilitate our model's 's transfer real data. See Fig. 2 We generate 49,430/2,700/2,,0 7,0/2,6000 such samples for the train/val/test splits, respectively. See Supp. materials for examples and details.
Real data collection. To explore whether models trained in simulation also work in the real world, we also collect set of real images and speech recordings while preserving the ground truth anechoic audio. To collect image data, we use iPhone 11 camera to capture panoramic RGB image and monocular depth estimation algorithm (Godard et al. 2019) to generate the corresponding depth image. To record the audio, we use ZYLIA ZM-1 microphone. For the source speech, we play utterances from the LibriSpeech test set through loudspeaker held by person facing the camera. We collect data from varying environments- aattronnnnnnt-addiiiiuunn meeting rooms, atriums, corridors, and classrooms. For each environment, we vary the speaker location from near-field to mid-field to far-field. For each location, we play utterances of diffcrent genders and speakers. During data collection, the microphone also records ambient sounds like people chatting, door opening, AC humming, etc. In total, we collect 200 recordings. Please see the Supp. for examples. We will publicly share all data and code.
5 APPROACH
We propose the Visually-Informe Dereverberation of Audio (VIDA) model, which leverages visual cues learn representations of the environmental acoustics and sound source locations derever- berate audio. While our model is agnostic to the audio source type, we focus on speech due to the importance of dereverberating speech for downstream analysis. VIDA consists of main compo- nents (Figure 3): 1) a Visual Acoustics Network (VAN), which learns to map RGB-D images of the environment features useful for dereverberation, and 2) the dereverberation module itself, which is based on UNet encoder-decoder architecture. The UNet encoder takes as input a reverberant spectrogram, while the decoder takes the encoder's output along with the visual dereverberation features produced by the VAN and reconstructs dereverberated version of the audio.
Visual Acousties Network. Visual observations of scene reveal information about room acoustics, including room geometry, materials, object locations, and the speaker position. We devise the VAN to capture all these cues into a latent embedding vector, which is subsequently used to remove reverb. This network takes as its input an RGB image I, and depth image Ig. captured from the listener's s current position within the environment. The depth image contains information about the geometry of the environment and arrangement of objects, while the RGB image contains information about
5

---

Under review as a conference paper at ICLR 2022
their material composition. To better model these different information sources, we two separate ResNet18 (He et al. 2016) networks to extract their features, i.c. e, = f., and e = fa(Ia). We concatenate e, and ea channel-wise and feed the result to a Ixl convolution layer fe(-) to reduce the number of total channels to 512 followed by a subsequent pooling layer fi(-) to reduce the spatial dimension, resulting in the output vector e. f((fe((e;;ea)))
Dereverberation Network. To recover the clean speech audio, we use the UNet (Ronneberger al. 2015) architecture, a fully convolutional network often used for image segmentation. We first use the Short-Time Fourier Transform (STFT) to convert the reverberant input audio A, to a complex spectrogram Sy. We treat S, as 2-dimensional, 2-channel image, where the horizontal dimension represents time, the vertical dimension represents frequency, and the two channels represent the log-magnitude and phase angle. Our UNet takes spectrograms of fixed size of 256 x 256 as input, but in general the duration of the speech audio we wish to dereverberate will be variable. Therefore, the model processes the full input spectrogram using a series of overlapping. sliding windows. Specifically, we segment the spectrogram along the time dimension into sequence of fixed-size chunks 5: = {S,, S2, ..., ST } using a sliding window of length s frames and 50% overlap between consecutive windows to avoid boundary artifacts. To derive the ground-trute -uuut- target spectrograms used in training, we perform the exact same segmentation operation on the clean source audio Ag to obtain Sge (S. 5 ..., S..
During training. when a particular waveform S, is selected for inclusion in a data batch, we randomly sample one of its segments S be the input to the model, and choose the corresponding St as the target. We first compute the output of the VAN, ec. for the environment image associated with S,. Next, S is fed to the UNet's encoder extract the intermediate feature map ex fene (S,), We then spatially tile and concatenate ef depth-wise with e", and feed the fused features to the UNet decoder,
which predicts the source spectrogram segment S = fdee((e", e*]).
Spectrogram prediction loss. The primary loss function we use to train our model is the Mean- Squared Error (MSE) between the predicted and ground-trutt -tuth spectrograms, treating the magnitude and phase separately. For given predicted spectrogram segment Sy. let Mj denote the predicted log-magnitude spectrogram, P denote the predicted phase spectrogram, and M; and P denote the respective moound-rruth magnitude and phase spectrograms. We define the magnitude loss as:
Lmeenstude = ||M! - M2ll2
(1)
To address the issue of phase wraparound, map the phase angle to its corresponding rectangular coordinates on the unit circle and then compute the MSE loss for the phase:
Lpaase = || sin (P:) - sin( (a)(P + cos( - cos( (p))
(2)
Reverb-visual matching loss. To reinforce the consistency between the visually-inferred room acoustics and the reverberation characteristics learned by the UNet encoder, we also employ a contrastive reverb-visual matching loss:
Lmatching (e", ee. ee = -aa + 0,.
(3)
Here, d(z, y) represents L2 distance, f-(()) applies L2 normalization, m is a margin, and e, is different speech embedding sampled from the same data batch. This loss forces the embeddings output by the VAN and the UNet encoder to be consistent, which we empirically show to be beneficial.
Training. Our overall training objective (for single training example) is as follows:
Ltotal Lmagnitude + Ly + >g
(4)
where Ar and A2 are weighting factors for the phase and matching losses. To augment the data, we further choose rotate the image view for random angle for each input during training. This is possible because our audio recording is onni-directional and is independent of camera pose. This data augmentation strategy prevents the model from overfitting; without our model fails to converge. This strategy creates a one-to-many mapping between reverb and views, forcing the model to learn tiewpoint-invarte representation of the room acoustics.
Testing. At test time, we wish to re-synthesize the entire clean waveform instead of single fixed- length segment. In this case, we feed all of the segments for a waveform S, into the model and temporally concatenate all of the output segments. Because consecutive segments overlap by 50%, during the concatenation step we only retain the middle 50% of the frames from each segment and discard the rest. Finally, re-synthesize the waveform we the Griffin-Lim algorithm (Griffin &

---

Under review as a conference paper at ICLR 2022
Speech Enhancement PESQ f
Speech Recognition WER (%) WER-FT (%) 4
Speaker Verification
EER (%) 4 EER-FI (%) 4
Anechoic (Upper bound)
4.64
2.50
2.50
1.62
1.62
Reverberant
1.54
8,86
4.62
4.69
4.57
MetricGAN+ Fu et al. 2021
2.33 (+51%)
7.49 (+15%)
4.86 (-5%)
4.67 (-0.4%)
2.75 (+39%)
HiFi-GAN (Su ct al
2020b
1,83 (+19%)
9.31 (-5%)
5,59 (-20%)
4.30 (48%)
2.49
(+46%)
WPE (Nakatani al 2010)
1.63 (+6%)
8.18 (+8%)
4.30 (+7%)
5.19 (-11%)
4.48 (+2%)
Audio-only dereverb.
2.32 (+51%)
4.92 (+44%)
3.76 (+19%)
4.67 (-0.4%)
2.61
(+43%)
VIDA w/ normal FoV
2.33 (+51%)
4.85
(+15%
3.73
(+19%)
4.53 (+3%)
2.79
(+39%)
VIDA w/o matching loss
2.38 (+55%)
4.59
(+18%
3.72 (+19%)
4.02 (+14%)
2.62 (+43%
VIDA w/o human mesh
2.31
(+50%)
4.57
(+18%)
3.72 (+19%)
4,00 (+15%)
2.52
(+15%)
VIDA w/ random image
2.34
(+5286)
4.94
(+44%)
3.82
(+17%)
4,70 (-0.2%)
2.48 (+25)
VIDA
2.37
(+54%)
4,44
(+50%)
3.66
(+21%)
3.97 (+15%)
2.40
(+47%)
Table I: Results multiple speech analysis tasks, evaluated on the LibriSpeech test-clean set that is reverberated with our environmental simulator (with the exception of the "Anechoic (Upper bound)" setting, which is evaluated on the original audio). WER-FT and EER-FT refer tests where the ASR and verification models are finetuned with the audio-enhanced data, respectively. The relative improvement compared to Reverberant included in parentheses.
Lim 1984) to iteratively improve the predicted phase for 30 iterations, which we find works better than directly using the predicted phase or using Griffin-Lim with randomly initialized phase.
6 EXPERIMENTS
We evaluate our model by dereverberating speech for three downstream tasks: speech enhancement (SH), automatic speech recognition (ASR), and speaker verification (SV). We evaluate using both real scanned Matterport3D environments with sim audio as well as real-world data collected with camera and mic. Please see Supp. for all hyperparameter settings and data preprocessing details.
Evaluation tasks and metrics. We report the standard metrics Perceptual Evaluation of Speech Quality (PESQ) (Rix et al. 2001). Word Error Rate (WER), and Equal Error Rate (EER) for the three tasks, respectively. For ASR and SV, we use pretrained models from the SpeechBrain (Ravanelli al. 2021) toolkit. We evaluate these models off-the-shelf on our (de)reverberated version of the LibriSpeech test-clean set, and also explore finetuning the model on the (de)reverberated LibriSpeech train-clean-366 data. For performing speaker verification, we construct a set of 80k sampled utterance pairs consisting of different rooms, mic placements and genders to account for session variability, similar to Richey et al. (201)). Please see Supp. for more details.
Baseline models. In addition to evaluating the anechoic and reverberant audio (with enhance- ment), we compare against multiple baseline dereverberation models: 1. MetricGAN+ (Fu et al.) 202T): a state-of-the-art model for speech enhancement; we use the public implementation from SpeechBrain (Ravanelli et al. (02T), trained on our dataset. Following the original paper, we optimize for PESQ during training, then choose the best-performing model snapshot (on the validation data) specific to each of the downstream tasks. 2. HiFi-GAN (Su et al. (020b): a recent model for denoising and dereverberation WPE (Nkataan et al., 2010): widely used statistical speech dereverberation model. 4. Audio-only dereverb: An ablation of VIDA that does not use any visual input or the proposed matching loss (i.e., the VAN removed), uses only the U-Net trained with the MSE loss for dereverberation; similar model is proposed by Ernst et al. (2018).
We emphasize that all baselines are audio-only models, as opposed to our proposed audio-visuee model. Our multimodal dereverberation technique could extend to work in conjunction with other cowly-proposed audio-only models, i.c., ongoing architecture advances are orthogonal to our idea.
Results in scanned environments. Table [ shows the results for all models on SE, ASR, and SV. First, since existing methods report results on anechoic audio, we note the pretrained SpeechBrain model applied to anechoic audio (first row) yields errors competitive with the SoTA (Gulati et al.) 2020), meaning we have a solid experimental testbed. Comparing the results on anechoic vs. re- verberated speech,
that reverberation significantly degrades performance on all tasks. Our VIDA model outperforms all other models, and by a large margin on the ASR and SV tasks: without We use this implementation:
Figan- danoi ser

---

Under review as a conference paper at ICLR 2022
Speech Enhuncement PESQ t
Speech Recognition WER (%)
Speaker Verification EER (%)
Anechoic (Upper bound)
4.64
2.52
1.42
Reverberant
1.22
18.39
3.91
MetricGAN+ (uu et al. 2021
1.62 (+33%)
21.42 (-16%)
5.70 (-46%)
HiFi-GAN (Su et al. 20206
1.33 (-9%)
24.05 (-31%)
5.21 (-33%)
Audio-only dereverb.
1.41 (+16%)
15.18 (+17%)
4.24 (-8%)
VIDA w/ normal FoV
1.44 (+18%)
14.71
(+20%)
3.79 (+39)
VIDA
1.49 (+22%)
13.02
(+29%)
3.75 (+4%)
Table 2: Results on real data demonstrating sim2real transfer. Gain vs. Reverberant in parens.
Atrium
Auditorium
Meeting Room
Classroom
Corridor
Near-ficld (1-3m)
14.10/ 8.97
0.91 / 0.91
4.98 / 6.47
6.14 5.26
2.15 1.79
Mid-field (3-8m)
21.78 18.94
5.06 / 6.32
7.67 / 7.67
2.56 / 1.47
7.27 / 4.36
Far-field (8+m)
52.38 50,52
10.44 / 7.46
21.95 / 6.71
5.91
25.23 / 21.10
Table 3: Breakdown of word error rate (WER) for Audio-only dereverb. / VIDA on real test data.
finetuning, we achieve absolute improvements of 0.04 PESQ (1.71% relative improvement), 0.48% WER (9. 9.75% relative improvement,, and 0.68% EER (14.56% relative improvement) over the best baseline in each case (which happens to be the audio-only version of VIDA for both the ASR and SV tasks). The results are statistically significant according to a paired t-test (p-values are 1.56e-60 for PESQ, 3.70e for WER, 2.58e-43 for speaker verification scores). After finetuning the ASR and SV models, the gains are still largely preserved at 0.1% WER (2.66% relative), and 0.21% 1% EER (8.0 88.03% relative), although it is important to note that finetuning downstream models on enhanced speech is not always feasible, e.g., if using an off-the-shelf ASR system. Our results demonstrate that learning the acoustic properties of environment from visual signals is very helpful for dereverberating speech, enabling the model to leverage information unavailable in the audio alone.
To understand how the WER of our model changes the input difficulty increases, plot the cumulative WER against (a) the PESQ of the input speech and (b) the speaker distance from the camera in Fig. and Fig. [bb] respectively. Our VIDA model consistently outperforms the Audio-only baseline and MetrieGAN+ (Fu et al. 2021) on all difficulty levels. Importantly, when the input sample more difficult (low PESQ or far distance), our model shows greater performance gain, Ablations. To understand how well VIDA works with a normal field-of-view (FoV) camera, we replace the panorama image input with a FoV of 80 degrees randomly sampled from the current view. Table shows the results. All metrics drop compared to using a panorama. This is expected, because the model is limited in what it can see with narrower field of view; the inferred room acoustics are impaired by not seeing the full environment or missing where the speaker is. Compared to the audio-only dereverberation model, however, VIDA still performs better; even a partial view of the environment helps the model understand the scene and dereverberate the audio.
Next, we ablate the proposed reverb-visus matching loss ("w/o matching loss"). Without it, VIDA's performance declines all metrics. This shows by forcing the visual feature to agree with the reverberation feature, our model learns a better representation of room acoustics.
To examine how much the model leverages the human speaker cues and uses the visual scene, we evaluate our trained VIDA model on the same test data but with the 3D humanoid removed ("w/o human mesh") or train VIDA with random images ("w/ random image") and evaluate. All three metrics become worse. This shows our model pays attention to both the presence of the human speaker and the scene geometry to learn a better representation of the anticipated reverberation.
Lastly, to understand accurately estimating room acoustics characteristics (c.g. RT60) is enough for dereverberation, we equip the audio-only dereverb baseline with predicted room acoustics charac- teristics and show that these models still underperform VIDA. See details in Supp.
Robustness to noise and ambient sounds. We test the robustness of our model by adding ambient sounds from environments such as coffee shops and restaurants, using the WHAM dataset (Wichern) et al.) 2019). We add them to the reverberant test waveform with 20dB SNR, following (Ernst et al. 2018 Nakatani et al. 2010). We show our model is robust to this noise; see Supp.
Results on real data. Next, we deploy our model in the real world. We use all models trained in simulation dereverberate the real-world dataset (cf. Sec. 4 before using the finetuned ASR/SV
8

---

Under review as a conference paper at ICLR 2022
GAN Audio-onlly
Audio only
WER
WEN
242
Vibble
62
Inpul PESO)
Distance (m)
embedding
0.0
Visual emriedding
(a) WER PESQ
(b) WER vs distance
(c) TSNE for RT60
(d) TSNE for room size
Figure 4: Left: Cumulative WER against PESQ of the imput speech (a) and the speaker distance.
from the camera (b). Right: TSNE visual features colored by RT60 (c) and visual room size (d).
Parorama Imago
Reverbbrantt
Figure 5: Example input images, anechoic spectrograms, reverberant spectograms and spectrograms dereverberated by VIDA (top from a scan, bottom a real pano). The speaker is out of view in the first case and distant in the second case (back of the classroom). Though both received audio inputs are quite reverberant, our model successfully removes the reverb and restores the source speech.
models evaluate the enhanced speech. Table shows the results of models on real data. Reverberation does more damage to the WER compared to in simulation. Although MetrieGAN+ (Fu jtt al. 202T) has surprisingly better PESQ. it has a weak WER score. Our VIDA model again outperforms all baselines on ASR and SV. This demonstrates the realism of the simulation and the capability of our model to transfer to real-world data, a promising step for VIDA's wider applicability.
Table 3aleeBreeee breaks down the ASR performance for Audio-only dereverb. (the best baseline) and our VIDA model by environment type and speaker distance. The atrium is quite reverberant due to the large space. Although the auditorium is similarly large, the space is designed reduce reverberation and thus both models have lower WER. The meeting room and the classroom have smaller sizes and are comparatively less reverberant. The corridor only becomes reverberant when the speaker is far away. VIDA outperforms the Audio-only dereverb. baseline in most cases, especially in highly reverberant ones. See the Supp. video for examples.
Qualitative examples. Figure shows a simulated and reverberant spectrogram is much blurrier compared to the anechoic spectrogram, while our predicted spectrogram removes those reverberations by leveraging the visual cues of room acoustics.
example. As we can see, the
Analyzing learned visual features. Figure 4c and 4d analyze our model's 's learned visual features 2D TSNE projections (van der Maaten Hinton 20088. For each sample, we color the point according to either (c) RT60 or (d) visible room volume from the camera's 's viewpoint calculated based on the depth map (depth values truncated at 10m). Neither of these variables are available to our model during training, yet when learning to perform deverberation, our model exposes these high-level properties relevant to the audio-visual task. Consistent with the quantitative results above, this analysis shows how our model captures elements of the visual scene, room geometry, and speaker location that are valuable to proper dereverberation.
CONCLUSION
We introduced the novel task of audio-visual dereverberation. The proposed VIDA approach learns to remove reverb by attending to both the audio and visual streams, recovering valuable signals about room geometry, materials, and speaker locations from visual encodings of the environment. In support of this task, we develop a large-scale dataset providing realistic, spatially registered observations of speech and 3D environments. VIDA successfully dereverberates novel voices in novel environments more accurately than an array of baselines, improving multiple downstream tasks. Trained a simulator, model also shows promise enhancing speech in real-world data, though more work is needed to bring down the absolute enror rates. In future work, we will explore temporal models for dereverberation with real-world video and introduce active perception for camera control.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
We provide the training details in Sec. 5 implementation details (hyperparameters and architectures) in Sec. AA.3] and evaluation details in Sec. A.4 For dataset creation, details can be found in Sec. 2 Both the code and dataset will be made public if accepted.
ETHICS STATEMENT
The research this paper does not involve human subjects. The people the real data have consented to share their imagery. Our datasets- both the simulations and those images be shared publicly, All authors of this paper follow the ICLR Code of Ethics (htp:://cch.ccpphbiiiccooooohhss
REFERENCES
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech enhancement. In INTERSPEECH, 2018.
Triantafyllos Afouras, Andrew Owens, Joon-Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. In ECCV, 2020.
Jont B. Allen and David A. Berkley. Image method for efficiently simulating small-room acoustics. The Journal of the Acoustical Society of America, 65(4):943-950, 1979.
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran.
Self-supervised learning by cross-modal audio-video clustering. In NeurlPS, 2020.
Andrew Zisserman Arsha Nagrani, Joon Son Chung. Voxceleb: large-scale speaker identification dataset. In INTERSPEECH, 2017.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. 3DV, 2017. MatterPort3D dataset license available at: mn..tum.de/matee
/MP _OP_ pdf
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Kr- ishna Ithapu, Philip Robinson, and Kristen Grauman. SoundSpaces: Audio-visual navigation in 3d environments. In ECCV, 2020.
Victoria Dean, Shubham Tulsiani, and Abhinav Gupta. See, hear, explore: Curiosity via audio-visual association. In NeurIPS, 2020.
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TINN emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Helen Meng, Bo Xu, and Thomas Fang Zheng (eds.), Interspeech 2020, pp. 3830-3834. ISCA, 2020.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. NIPS, 2014.
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. In SIGGRAPH, 2018.
Ori Ernst, Shlomo E. Chazan, Sharon Gannot, and Jacob Goldberger. Speech dereverberation using fully convolutional networks. 2018.
Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. MetricGAN: Generative adversarial networks based black-box metric scores optimization for speech enhancement. 2019,
Szu-Wei Fu, Cheng Yu, Asun-An Hsich, Peter Plantinga, Mirco Ravanelli, Xugang Lu, and Yu Tsao. MetrieGAN+: An improved version of MetricGAN for speech enhancement. 2021.
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In ICRA, 2020.
10

---

Under review as a conference paper at ICLR 2022
Ruohan Gao and Kristen Grauman. 2.5d visual sound. In CVPR, 2019.
Ritwik Giri, Michael Seltzer, Jasha Droppo, and Dong Yu. Improving speech recognition in reverberation using room-awan deep neural network and multi-task learning. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2014-5018. IEEE, 2015.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self- supervised monocular depth prediction. In ICCV, 2019.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions acoustics, speech, and signal processing, 32(2):236-243, 1984.
Anmol Gulati, Chung Cheng Chiu, James Qin, Jiahui Yu, Niki Parmar, Ruoming Pang, Shibo Wang, Wei Han, Yonghui Wu, Yu Zhang, and Zhengdong Zhang. Conformer: Convolution-augmennte transformer for speech recognition. In Interspeech, 2020.
Kun Han, Yuxuan Wang, DeLiang Wang, William S. Woods, Ivo Merks, and Tao Zhang. Learning spectral mapping for speech dereverberation and denoising. 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. In ICASSP, 2016.
Martin Holters, Tobias Corbach, and Udo Zölzer. Impulse response measurement techniques and their applicability in the real world. In Proceedings of the 12th International Conference on Digital Audio Effects, DAFx 2009, 2009.
Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Was Chang, and Hsin-Min Wang. Audio-visual speech enhancement using multimodal deep convolutional neural networks. In IEEE Transactions on Emerging Topics in Computational Intelligence, 2017.
Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching, In NeurIPS, 2020.
Marco Jeub, Magnus Schäfer, Hauke Krüger, Christoph Matthias Nelke, Christophe Beaugeant, and Peter Vary. Do we need dereverberation for hand-held telephony? In International Congress on Acoustics, 2010.
Yuri Khokhlov, Alexander Zatvorittkkiy, Ivan Medennikov, Ivan Sorokin, Tatiana Prisyach, Aleksei Romanenko, Anton Mitrofanov, Vladimir Bataev, Andrei Andrusenko, Mariya Korenevskaya, and Oleg Petrov. R-vectors: New technique for adaptation to room acoustics. In INTERSPEECHI 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Keisuke Kinoshita, Mare Delcroix, Sharon Gannot, Emanuël Habets, Reinhold Haeb-Umbace Walter Kellermann, Volker Leutnant, Roland Maas, Tomohiro Nakatani, Bhiksha Raj, Armin Sehr, and Takuya Yoshioka. A summary of the reverb challenge: State-of-the-art and remaining challenges in reverberant speech processing research. EURASIP Journal on Advances in Signal Processing, 2016.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5220-5224 IEEE, 2017.
Homare Kon and Hideki Koike. Estimation of late reverberation characteristics from single two- dimensional environmental image using convolutional neural networks. Journal of the Audio Engineering Society, 2019.
11

---

Under review as a conference paper at ICLR 2022
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from sulf-suee synchronization. In NeurlPS, 2018.
Dingzeyu Li, Timothy R Langlois. and Changxi Zheng. Scene-aware audio for 360 videos. ACM Transactions on Graphics (TOG), 37 2018.
Sagnik Majumder. Ziad Al-Halah, and Kristen Grauman. Move2hear: Active audio-visual source separation. In ICCV, 2021.
Peter Vary Marco Jeub, Magnus Schafer. A binaural room impulse response database for the evaluation of dereverheration algorithms. Proceedings of International Conference on Digital Signal Processing, 2009.
Daniel Michelsanti, Zheng-Hur Hua Tan, Shi Xiong Zhang, Yong Xu, Meng Yu, Dong Yu, and Jesper Jensen. An overview of dlep-learningghassee audio-visual speech enhancement and separation. In arXiv, 2020.
Masato Miyoshi and Y. Kaneda. Inverse filtering of room acousties. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1988.
Pedro Morgado, Nono Vasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. In NeurlPS, 2018.
Pedro Morgado, Yi Li, and Nuno Vasconcelos. Learning representations from audio-visual spatial alignment. In NeurIPS, 2020.
Damian T Murphy and Simon Shelley. Openair: An interactive auralization web resource and database. In Audio Engineering Society Convention 129, 2010
D.T. Murphy, Antti Kelloniemi, Jack Mullen, and Simon Shelley. Acoustic modeling using the digital waveguide mesh. Signal Processing Magazine, IEEE, 24:55 66. 04 2007.
Tomohiro Nakatani, Takuya Yoshioka, Keisuke Kinoshita, Masato Miyoshi, and Biing-Hwang Juang. Speech dereverberation based on variance-normaalzze delayed linear prediction. In IEEE Transactions on Audio, Speech, and Language Processing, 2010.
Patrick A. Naylor and Nikolay D. Gaubitch. Speech Dereverberation. Springer Publishing Company, Incorporated, 1st edition, 2010.
Stephen T. Neely and Jont B. Allen. Invertibility of room impulse response. In Journal of the Acoustical Society of America, 1979.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with supe-supe multisensory features. In ECCV, 2018
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210 IEEE, 2015.
Mirco Ravanelli, Titouan Parcollet, Aku Rouhe, Peter Plantinga, Elena Rastorgueva, Loren Lugosch, Nauman Dawalatabad, Chou Ju-Chieh, Abdel Heba, Francois Grondin, William Aris, Chien- Feng Liao, Samuele Cornell, Sung-Lin Yeh, Hwidong Na, Yan Gao, Szu- Fu, Cem Subakan, Renato De Mori, and Yoshua Bengio. Speechbrain. https: /tt:://gittt /gggttbbbb./g/o////////g com/ speechbrain/ speechbrain, 2021.
Colleen Richey, Maria A. Barrios, Zeb Armstrong, Chris Bartels, Horacio Franco, Martin Graciarena, Aaron Lawson, Mahesh Kumar Nandwana, Allen Stauffer, Julien van Hout, Paul Gamble, Jeff Hetherly, Cory Stephenson, and Karl Ni. Voices obscured in complex environmental settings (voices) corpus, 2018.
Antony W. Rix, John G. Beerends, Michael P. Hollier, and Andries P. Hekstra. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings, 2001.
12

---

Under review as a conference paper at ICLR 2022
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-24121111 Springer. 2015.
Mostafa Sadeghi, Simon Leglaive, Xavier Alameda-PInede Laurent Girin, and Radu Horaud. Audio- visual speech enhancement using conditional variational auto-encoders. In /EEE/ACM Transactions on Audio, Speech and Language Processing, 2020.
Nikhil Singh, Jeff Mentch, Jerry Ng. Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-modal reverb impulse response synthesis. In ICCV, 2021.
David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X- vectors: Robust DNN embeddings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5329-5333. IEEE, 2018.
Guy-Bart Stan, Jean-Jacques Embrechts, and Dominique Archambeau. Comparison of different impulse response measurement techniques. journal of the audio engineering society, 50(4): 249-262, april 2002.
Daniel Stoller, Sebastian Ewert, and Simon Dixon. Adversarial semi-supervised audio source separation applied to singing voice extraction. In ICASSP, 2018.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, al. The replica dataset: digital replica of indoor spaces. arXiv preprint arXiv: arXiv:/906.008 1906.05797 2019.
Jiaqi Su, Zeyu Jin, and Adam Finkelstein. Acoustic matching by embedding impulse responses. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 426 430. IEEE, 2020a.
Jiaqi Su, Zeyu Jin, and Adam Finkelstein. HiFi-GAN: High-fidelity denoising and dereverberation based on speech deep features adversarial networks. INTERSPEECH, 2020b.
Igor Szöke, Miroslav Skácel, Ladislav Mošner, Jakub Paliesek, and Jan Černocky. Building and evaluation real room impulse response dataset. IEEE Journal of Selected Topics in Signal Processing, 2019.
Ke Tan, Yong Xu, Shi-Xiong Zhang. Meng Yu, and Dong Yu. Audio-visual speech separation and dereverberation with a two-stage multimodal network. In IEEE Journal of Selected Topics in Signal Processing, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing hign-dimensional data using t-sne. In Journal of Machine Learning Research, volume 9, pp. 2575,2605, 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Shinji Watanabe, Michael I. Mandel, Jon Barker, and Emmanuel Vincent. Chime-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings. CoRR, abs/2004.09249, 2020.
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. In Proc. Interspeech, 2019.
Bo Wu, Kehuang Li, Minglei Yang, and Chin-Hui Lee. A reverferation-time-aaar approach to speech dereverberation based on deep neural networks. In IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2016.
Ruilin Xu. Rundi Wu, Yuko Ishiwaka, Carl Vondrick, and Changxi Zheng. Listening to sounds of silence for speech denoising. In NeurIPS, 2019.
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV, 2019a.
13

---

Under review as a conference paper at ICLR 2022
Yan Zhao, Zhong- Qiu Wang, and DeLiang Wang. Two-stage deep learning for noisy-reverberant speech enhancement. JEEE/ACM Trans. on Audio, Speech, and Language Processing. 2019b.
Yan Zhao, DeLiang Wang, Buye Xu, and Tao Zhang. Monaural speech dereverberation using temporal convolutional networks with self attention. 2020.
Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo, and Xiaogang Wang. Vision-infused deep audio inpainting. In ICCV, 2019.
14

---

Under review as a conference paper at ICLR 2022
A SUPPLEMENTARY MATERIALS
In this supplementary material, we provide additional details about:
1. Video (with audio) for demos the collected data as well as qualitative assessment of VIDA's performance.
Distribution of average RT60 over room types.
3. Implementation details of our model and data pre-pro prepprocessing
4. Evaluation details of downstream tasks and corresponding metrics.
5. Robustness to noise and ambient sound. Predicting room acoustics characteristics.
Ablation on visual sensors.
8. Cumulative WER against PESQ and distance.
A.1
QUALITATIVE VIDEO
This video includes examples for audio-visee visual data in simulation and in the real-world. We demon- strate examples of our dereverbration model applied to these inputs.
A.2 RT60 DISTRIBUTION
Table shows the distribution of average RT60 over some common room types in the Matterport3D training environments. Different rooms have different room acoustics characteristics, associated with their room sizes, typical object arrangements and materials. Importantly, those properties visually detectable. For example, bedrooms are usually smaller than garage or lounge, and have beds with absorptive materials. The joint effects of these factors lead small reverberation time. This knowledge is what models could learn to leverage for understanding room acoustics and performing dereverberation better.
Bedroom
Dining Room|
Hallway
Living Room
Garage
Lounge Conference Room
RT60
0.211s
0.224s
0.239s
0.242s
0.255s
0.318s
0.393s
Table 4: Distribution of average RT60 when the speaker is Im away from mic.
A.3 IMPLEMENTATION DETAILS
For the STFT calculation, we sample the input audio at 16 kHz and use a Hamming window of size 400 samples (25 milliseconds), hop length of 160 samples (10 milliseconds). and a 512 512-point FFT. By retaining only the positive frequencies and segmenting the spectrograms into 256- framee chunks (corresponding approximately 2.5 seconds of sound), the final audio input size our U-Net is 256x256. We use the Adam optimizer (Kingma & Ba, 2015) to train our model with Ir = 0.001. We decay the learning rate exponentially to 0.0001 in 150 epochs. We the batch size 96 and train all models for 150 epochs, which is long enough to reach convergence. We set the margin to 0.5, phase loss weight λ₁ to 0.08 and matching loss weight X2 0.001.
A.4 EVALUATION DETAILS
We evaluate our model on three tasks: speech enhancement (SE), automatic speech recognition (ASR), and speaker verification (SV).
For SE, the goal is improve the overall sonic quality of the speech signal, which measure automatically using the standard Perceptual Evaluation of Speech Quality (PESQ) (Rix et al. 2001) metric.
For ASR, the goal is to automatically transcribe the sequence of words that were spoken in the audio recording. For this task, we report the Word Error Rate (WER), which the standard metric used in ASR and reflects a word-level edit distance between recognizer's output and the ground-trun truth transcription.
For SV, the goal is to detect whether or not two different spoken utterances were spoken by the same speaker. For SV, report the Equal Error Rate (EER), standard metrie in the SV field indicating the point on the Detection Error Tradeoff (DET) curve where the false alarm and missed detection probabilities are equivalent.
15

---

Under review as a conference paper at ICLR 2022
Speech Enhancement PESQ f
Speech Recognition WER (%) + WER-FT (%)
Speaker Verification
EER (%) 4 EER-FT (%) 4
Anechoic (Upper bound)
4.64
2.50
2.50
1.62
1.62
Reverberant
1.36
12.27
6,38
4.69
5.10
MetricGAN+ (uu et al. 2021
2.12 (+57%)
9,40 (+23%)
7.09 (-11%)
4.94
(-5%)
3.38
(+34%)
HiFi-GANN (Su et all 2020b)
1,55 (-14%)
11,90 (43%)
7,80 (%)
5.18
(-10%)
3.80 (+25%)
WPE (Nakatani et al. 2010)
1.39 (42%)
11.32 (+8%)
7.00 (-10%)
4.48 (+4%)
4.95 (43%)
Audio-only dereverb.
1.76 (+29%)
7.37 (+40%)
5.52 (+14%)
5.75 (-23%)
3.58 (+30%)
VIDA w/ normal FoV VIDA w/o matching loss
1.76 (+29%)
7.51 (+39%)
5,51 (+14%)
5.54
(-18%)
3.40 (+33%)
1.81 (+33%)
6.76 (+45%)
5,31 (+17%)
4.95 (-6%)
3.26 (+36%)
VIDA
1.82 (+34%)
6,53 (+47%)
5.29
(+17%
4.83 (-3%)
3.13 (+39%)
Table 5: Results on the LibriSpeech test-clean set mixed with ambient sounds at a 20 dB signal-to- noise ratio.
Since the spectrogram MSE loss we optimize during training does not perfectly correlate with these three task-specific metrics, we perform model selection (across snapshots saved each training epoch) by computing the task-specific evaluation metric on 500 validation samples. We then select the best model snapshot independently for each downstream task and evaluate on the held-out test set; the same model selection procedure is also used for all of our baseline models.
For the ASR and SV tasks, we use the SpeechBrain (Ravanelli et al.) 2021) toolkit. For ASR, we use the HuggingFace Transformer (Vaswani et al., 2017) Transformer LM model pre-trained on LibriSpeech (Panayotov et al. 2015). We evaluate this model off-the-shelf on our (de)reverberated ver- sion of the LibriSpeech test-clean set, and also explore fine-tuning the model on the rdeerreeeeerated LibriSpeech train-clean- 360 data. For the SV task, we use SpeechBrain's 's EC -PANNDNN embedding model (Desplanques et al., 2020). pre-trained on VoxCeleb (Arsha Nagrami 2017). For performing verification, we evaluate the model on a set of 80k randomly sampled utterance pairs from the test-clean set (40k same-speaker pairs, 40k different--speaer pairs) using the cosine similarity-basee scoring pipeline from SpeechBrain's VoxCeleb recipe. In the verification task, we use the clean mnon-reverberated) speech as the reference utterance, and the reverberant speech as the test utterance. As in the ASR task, we evaluate this model on our dereverberation model's 's outputs both off- shelf, as well as after fine-tuning on the (de)reverberated train-clean-36 360 set.
A.5 ROBUSTNESS TO NOISY AUDIO
We test the robustness of our model by adding ambient sounds from urban environments such as coffee shops, restaurants, and bars using the WHAM dataset (Wichern et al. 2019). We add them to the reverberant test waveform with a SNR of 20, following (Ernst et al. 2018]] Nakatani et al., 2010 Table shows the results three downstream tasks. As expected, all models' performance drop compared to the results in the noise-free free test setting (Table 1), but our VIDA model still significantly outperforms the baselines ASR. For speaker verification, WPE (Nakatani et al. 2010) is reported to be robust to noisy input and thus has lower EER while using the pretrained model, but underperforms VIDA when the SV model is finetuned on the enhanced speech. Noise has less impact on the performance on MetricGAN+ et al. 2021) likely because it directly optimizes PESQ
A.6 PREDICTING ROOM ACOUSTICS CHARACTERISTICS
RT60 and direct-to-reverberant ratio (DRR) are some common characteristics used for measuring room acoustics (Giri et al., 2015). To understand if estimating these parameters accurately is enough for dereverberation, we first performed predictions of RT60, DRR, and distance purely based on audio with a ResNet network. After converging, the average absolute errors for them are 3e-3s, 160 dB and 17.92m respectively. It shows predicting DRR distance is quite hard from audio alone. Next, we concatenate the predicted quantities as an extra channel of input to the audio latent feature and train these models with the same setup until convergence.
Table 6 shows the results. Even though these models leverage extra supervision for training that is not available VIDA, they still underperform our VIDA model and only marginally improve the performance of the audio-only model. Among these three predicted quantities, we can see predicting distance helps the most, which is difficult predict even supplied with ground truth, but is more predictable from the visual stream.
16

---

Under review as a conference paper at ICLR 2022
Speech Enhuncement PESQ 1
Speech Recognition WER (%) 4
Speaker Verification EER (%)
Reverberant
1.54
8.86
4.69
Vanilla Audio-only
2.32
(+51%)
4.92 (+44%)
4.67
(+0.4%)
Audio-only w/ RT60 prediction
2.36 (+53%)
4.75 (-46%)
4.67 (+7%)
Audio-only w/ DRR prediction
2.34 (+52%)
4.86 (-45%)
4.51 (+4%)
Audio-only w/ distance prediction
2.36 (+53%)
4.74 (-47%)
4.27 (49%)
VIDA
2.37 (+54%)
4,44 (+50%)
3.97 (+15%)
Table 6: Augmenting Audio-only dereverb. with predicted room acoustics characteristics.
Speech Enhancement Speech Recognition Speaker Verification
PESQ
WER (%) 1
EER (%)
Reverherant
1.54
8.86
4.69
Audio-only dereverb.
2.32 (+51%)
4.92 (+44%)
4.67
(40.4%)
VIDA w/o RGB
2.38 (+55%)
4.76 (+46%)
3.82 (+19%)
VIDA w/o depth
2.38 (+55%)
4.52 (-49%)
3,99 (+15%)
VIDA w/ carly fusion
2.38 (455%)
4.56 (+48.5%)
3,94 (+16%)
VIDA
2.37
(454%)
4.44 (450%)
3.97 (+15%)
Table 7: Ablations on visual sensors. Percentages in parenthesis represent relative improvements over the reverberant baseline.
Audio-only
30
MetricGAN
Audio-only
MetricGAN+
HIFi-GAN
HiFi-GAN
VIDA
VIDA
WER (%)
WER (%)
4
5
1.25
1.50
1.75
2.00
2.25
2.50
10
15
20
Input PÉSQ
Distance (m)
(a) WER PESQ
(b) WER vs distance
Figure Cumulative WER against PESQ of the input speech (a) and the speaker distance.
A.7 ABLATION ON VISUAL SENSORS
To understand the importance of each input sensor, we ablate the RGB and depth input as shown in Table 7 Dropping either RGB depth makes the WER worse. We hypothesize that this is because they contain distinct information for the learning of room acoustics. The depth image is better for capturing room geometry, while the RGB image better for capturing material and speaker location information.
In addition, we perform early fusion of RGB and Depth images by stacking them along the channel dimension (w/ early fusion in Table ? and use one ResNet18 (He et al.] 2016) model instead of two. This method also has worse WER, which validates our design choice extracting RGB and depth features separately.
A.8 CUMULATIVE WER AGAINST WER AND DISTANCE
Figure 6 shows the cumulative WER against PESQ of the input speech and the speaker distance from the camera in Fig. 66 and Fig. 6b respective... In this plot, we include the curves for HiFi-GAN (Su al. 2020b) well, which underperforms audio-only dereverb. MetricGAN+ and VIDA by large margin.
A.9 PARAMETER COUNT AND EFFICIENCY.
In Table g we show the number of parameters and how long it takes to do one forward pass with batch size on RTX 6000 GPU for each model. The results averaged over 2000 runs. Despite
17

---

Under review as a conference paper at ICLR 2022
MetricGAN
HiFI-GAN
WPE
Audio-only
VIDA
Parameter Count Inference time
1.9M 0.032s
42.6M
17.7M 0.005s
43.7M
0.032s
0.011s
0.009s
Table Parameter count and inference time for each model.
Speech Enhancement
Speech Recognition Speaker Verification
PESQ f
WER (%) +
EER (%)
Vanilla Audio-only
2.32
4.92
4.67
Audio-only R-vectors
2.23
5.23
4.82
VIDA
2.37
4.44
3.97
Table Augmenting Audio-only dereverb. with trained R-vectors.
having more parameters, VIDA runs comparably to Audio-only (ablation of VIDA) and faster than MetricGAN+, HiFi-GAN and WPE.
A.10 COMPARISON WITH R-VECTORS
R-vectors (Khokhlov et al. 2019 is recently proposed approach that trains room acoustics aware network in a self-supervieed fashion and uses the feature this network to help improve distant speech recognition. Since there is no public implementation of this work, implement this approach based on descriptions the paper. We train the network with 49,430 distinct RIRs, each convolved with 20 speech clips, and the test accuracy is 88%. When using this network for dereverberation, we tile and concatenate R-vectors with the audio features similar to the audio-visual concatenation and train the network similarly audio-only dereverb. We refer this model as audio-only dereverb. + R-vectors. Table shows the results of this model in comparison to other models. This acoustics-aware vector does not improve the performance our audio-only model.
18


Reviews:
[{"review_id": "xq0qZGU0fWv", "reviewer": "Reviewer_j5yj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper takes an interesting and novel approach to de-reverberation.\nThe paper is well written, and each component of the method is well justified.\nThe qualitative analysis (e.g. figure 4) is a nice touch, and helps shed some light on the behavior of the proposed method (and baselines).\nThat said, I do have a few questions and suggestions.\n\nMy primary criticism is in the reporting of the evaluations (tables 1, 2, 3).\nEach setting is summarized by a single average score, and the \"best\" number is written in bold with no discussion given to the distribution of scores on the test data or the practical meaning of score differences.\nThis may give readers an inflated sense of performance here: is PESQ=2.38 really any different from PESQ=2.37? 2.33?  (Likewise for WER, EER, etc.)\nThere is a brief mention of astronomically low p-values (1.56e-60?) which I expect are primarily driven by the size of the test set, and not the size of the effect.\nI would strongly encourage the authors to do a more responsible job of reporting their results: include some error bars, or some other indication of the spread of the data!\n\nThe proposed method takes an interesting approach to optimizing both magnitude and phase (eqs 1 and 2) for the output spectrogram.\nThe Griffin-Lim algorithm is then used to refine the phase estimate, with a vague statement of this working better than directly using predicted phase or GL with random initialization.\nWhile I won't ask for more experiments on this, I think the point deserves a bit more explanation than what is given here, especially since there is no ablation study on the phase reconstruction loss.  (Nor is there any discussion of hyper-parameter tuning for equation 4 beyond the settings reported in the appendix.)\n\n\nI found Figure 5 (qualitative examples) to be pretty difficult to parse, and I say this as a reader with (probably) above-average experience reading spectrograms.\nWith a bit of squinting I can see what the authors are getting at, but it would be easier to read if the spectrograms were A) larger, and B) plotted on a logarithmic frequency axis so that the low frequencies (where most of the action is) occupy more of the visual real estate.\n\n\nFinally, it was nice to see the results in table 3, which show quite a strong effect on the performance of the system depending on differences in distance and room geometry.\nNot much is said about the geometric differences in the environments though, either for real or simulated data.\nIn particular, it looks from figure 2 as though there may be configurations of speaker/listener that have no direct path, in which one might say that everything observed is \"reverb\".\nIf this is the case, it would be interesting to see how performance (table 1) differs (if at all) in the presence or absence of a direct path.\n\n", "summary_of_the_paper": "This paper describes an audio de-reverberation method which integrates (panoramic, rgb+depth map) visual input with a spectrogram U-Net to estimate a de-reverberated spectrogram and recover the original clean signal.\nThe proposed method is compared to several baseline (audio-only) de-reverberation methods on speech enhancement, recognition, and speaker verification problems using both synthetic and real data.\nSeveral ablations of the proposed method are compared, along with some quantitative error analysis investigating the effects of distance and environment.\nThe proposed method appears to consistently improve on speech recognition and speaker verification, and perform about on par with prior work on speech enhancement (as measured by PESQ).\n\n", "summary_of_the_review": "Overall, I enjoyed this paper!  I think some of the reporting could be done better, but this is a minor complaint.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "mMWXE_dlaSM", "reviewer": "Reviewer_NGNE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While authors suggest visual information is helpful, it isn't reflected in the results. In Table 1, if we compare the \"Audio-only dereverb.\" and \"VIDA\" results, the gains are very modest. The PESQ 2.32 to 2.37, WER-FT 3.76 to 3.66, and SID EER 2.61 to 2.40. This raises the question about the effectiveness of the proposed approach. I feel the majority of the dereverberation is still learned by the audio model. This should be investigated. \n\nOverall, the technical contribution is limited to a new dataset and the addition of ResNet for visual information processing. Similar approaches have been used in the audio-visual enhancement with lip information. ", "summary_of_the_paper": "In this paper, the authors introduce a novel audio-visual dereverberation approach. They propose a Visually-Informed Dereverberation of Audio (VIDA) model for dereverberation. The authors also create synthetic/simulated datasets and real-world data for experimentation. Finally, they show the impact of the proposed VIDA model on several speech tasks including, recognition, enhancement, and speaker verification. The results are encouraging. The main contribution of this work is the use of visual information as an auxiliary input for dereverberation. \n\n\n\n", "summary_of_the_review": "I believe the manuscript in its current form offers a very limited contribution to the research community, and the contribution of visual information in dereverberation is minimal. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "g6Fm3L8ircw", "reviewer": "Reviewer_NeeT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- first multi-modal (audio-visual) approach applied to dereverberation\n- paper is well organized and straightforward to follow\n- solid experiments with ablation studies and in-depth analyses\n- supplementary materials provide useful information not explained/described in the main text\n\nWeaknesses:\n- some of the key arguments are not sufficiently validated\n- contributions are marginal and limited\n- experiments and evaluation are not thorough\n\nThe key idea in this work is that visual input captured by a camera conveys information that characterizes room acoustics. With all the experiments and ablation studies, however, I don't believe that the authors provide sufficient information that support the aforementioned key hypothesis. The authors did take into account this matter in two ways: first, they included a reverb-visual matching loss term to penalize a matching between random speech and visual input, but little experiment or analysis was performed on this loss term - varying the weight or using different loss functions, for example. With or without using a reverb-visual matching loss was included in the ablation studies, but the effect was not significant compared with other methods. The performance was better with the audio-only model even in speech quality metric (PESQ). The second way is to use a random image instead of using the image that matches the room impulse response. In such scenario, the images does not carry any information relevant to the room acoustics, and thus the performance should be worse than or comparable to that of the audio-only method. However, the performance is better in general and is even best for some metric. Further experiments and deeper analysis are warranted to investigate this issue.\n\nAnother point that I hope to be added is the comparison of the model efficiency. Adding a visual embedding network increases the number of model parameters and computational costs. The proposed method is likely to work in real-time scenarios, and possibly on small devices, the efficiency should also be considered.\n\nLastly, I'd like to suggest to include R-vectors approaches as another baseline for comparison.", "summary_of_the_paper": "This paper presents a deep neural net-based dereverberation algorithm that uses both audio and video modalities. Based on the observation that a visual scene captured by a camera conveys information that is related to room characteristics, the authors propose a visually informed audio dereverberation method that aims to extract clean, anechoic speech from reverberant speech. In doing so, they first construct a large audio-visual dataset synthesized using a 3D simulator for real-world scanned environments and LibriSpeech data. They then train deep neural networks that take as inputs both visual data (RGB and depth images) and audio data (reverberant speech), and output clean speech. When training, two types of losses - one for clean speech spectrogram estimation and the other for reverb-visual matching - are used. Through the experiments with several downstream tasks for speech, they showed that the proposed audio-visual dererberation method outperforms the baseline models, both for synthetic and real-world test data.", "summary_of_the_review": "This paper proposes a multi-modal learning framework that is applied to speech dereverberation. The idea is interesting and somewhat novel, but the experimental validation is not thorough enough to support the key idea of reverb-visual matching nor to provide sufficient evidences that visual stream does convey information about room acoustics.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fZx5zXxIvl", "reviewer": "Reviewer_Tgq4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\nS1) To my knowledge, this is the first audio-visual approach to dereverberation. I think it is interesting to see how much visual information can help.\n\nS2) The proposed dataset seems quite useful. A lot of existing synthetic reverb for such audio tasks is generated using shoebox room simulators, so having more realistic environments seems quite useful. The accompanying data recorded in real environments using loudspeaker playback (such that the clean reference is available) is also quite useful, and I hope the authors plan to release a recipe for this data.\n\nS3) The paper includes a lot of ablations, which I really appreciate. Though the focus of the paper is on audio-visual approaches to the task, I'm really glad the paper evaluates audio-only versions of the model, as well as other components such as the matching loss.\n\nS4) The paper is clear and well-written.\n\n### Weaknesses:\n\nW1) I listened to the demos in the supplementary material, and I feel like I heard a fair amount of artifacts and distortion in some examples: for moderate reverb, the model predictions have a certain \"buzziness\" to them, and more extreme levels of reverb have a \"twanginess\" to them. Something that would make this paper stronger would be a subjective human listening evaluation, e.g. MUSHRA, which would measure these kinds of artifacts better than an objective measure like PESQ. Also, alternative loss functions may help reduce artifacts. E.g. L1 loss instead of L2 on spectrogram magnitude, or MSE on power-compressed spectrograms can provide more perceptually-relevant loss functions.\n\nW2) The processing method of using 2.56s frames, then concatenating the non-overlapping parts together seems suboptimal to me. Why not use e.g. a Hann window to stitch the predicted patches with overlap-add, or invert back to the time-domain and do overlap-add and train through this?\n\nW3) Where does the \"random speech embedding sampled from the data batch.\" come from? How is it computed? Is this a speaker ID embedding?\n\nW4) The improvements by adding visual information are marginal compared to the audio-only model. The audio-only model does perform better than the baselines, but degrades PESQ compared to other baselines on real data. To benchmark the audio-only baseline against the literature, it would be interesting to compare to competitive systems in e.g. the REVERB challenge. I'm also not very convinced that visual input really helps that much.\n\nW5) Table 1 would be improved by adding trainable parameter counts. Is the proposed model only doing better because it has more trainable parameters? WPE itself has no trainable parameters, since it's just a model-based EM algorithm.\n\nW6) The lack of noise in the training and eval data limits real-world applicability, but I understand the focus of the paper in only on dereverberation. I'm glad that the authors tested noise robustness with WHAM! data, but 20 dB is a pretty high SNR for noise robustness. It would be interesting to see how performance degrades with decreasing SNR.\n\n### Minor comments\n\nM1) I feel that this statement isn't quite right: \"R is a function of...the relative positioning of the speaker and the listener.\" I would argue that an RIR can be a function of the absolute positioning of the speaker and the listener. E.g. what if a source is close to a wall, versus in the the middle of a room?\n\nM2) For the real data, would it be possible to be more specific about the source-to-microphone distances used in the real recordings, beyond \"near-field to mid-field to far-field.\"?\n\nM3) Also concerning real data: would it be possible to more more specific for this statement? \"For each location, we play around 10 utterances.\". How many utterances? Gender balanced? Different speakers across rooms?\n\nM4) Nit: U-Net instead of UNet?\n\nM5) \"While our model is agnostic to the audio source type,\": depends on the training data.\n\nM6) In Tables 1 and 2, I would say \"anechoic speech (upper bound)\" instead of \"clean speech (upper bound)\", because to me \"clean\" suggests there's additive noise.\n\nM7) About the blind RT60/DRR/distance predictions: \"After converging, the average errors for them are 3e-3s, 160 dB and 17.92m respectively.\" Are these absolute or squared errors?", "summary_of_the_paper": "This paper examines an audio-visual approach for dereverberation, where dereverberation of speech is conditioned on RGB and depth images (either field-of-view or panoramic). It proposes a dataset for this task based on real-world 3D scans of homes, using Librispeech data. Real data is also used The model is based on a U-Net conditioned on embeddings extracted by a \"visual acoustics\" network doing direct spectrogram prediction. The method is evaluated using PESQ, WER for speech recognition, and EER for speaker verification on synthetic and real data. The audio-visual method is found to perform marginally better than an audio-only version of the model. The audio-only version of the model outperforms several baselines from the literature on synthetic data, with mixed results on real data.", "summary_of_the_review": "Overall, I vote for marginal acceptance. This is an interesting novel task (audio-visual dereverberation), and the proposed synthetic and real data are quite interesting and useful for the community. But the paper's results show that visual input provides marginal performance gain over the audio-only baseline. The audio-only baseline outperforms some benchmarks in terms of PESQ, WER for speech recognition, and EER for speaker verification on synthetic data, and WER and EER for real data, but I have concerns about the subjective listening quality of the demos, compared to other dereverberation algorithms I have listened to. Thus, I am concerned that this isn't a significant improvement over audio-only prior work, at least in terms of subjective listening quality.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the dereverberation problem from the audio-visual perspective.  The geometry of the environment is represented by RGB and depth images.  The authors propose a so-called visually-informed dereverberation of audio (VIDA) model and also create a dataset consisting of both synthetic and real data to verify the effectiveness of the model.  Experiments are conducted on speech enhancement, speech recognition and speaker identification tasks.  The authors compare VIDA with audio only dereverberation as well as various established baseline systems in the community.  

The audio-visual way of coping with dereverberation using visual representation of the acoustic environment seems to be interesting. The authors' rebuttal has cleared most of the concerns raised by the reviewers but there are still numerous lingering concerns which affect its acceptance.  First of all,  most of the reviewers consider the novelty not overwhelmingly significant.  Second, the contribution of the visual input seems to be only marginal compared to the audio-only dereverberation. Results on real data are also mixed.  Some of the reported p-values are extremely small, which raises questions whether it is due to the size of the test set.  Third, there are noticeable artifacts in some of the samples in the demo.  Fourth,  there are numerous issues in the paper that are worth further in-depth investigation. For instance, it would be helpful to show in which way exactly the RGB and depth images helps.
Decision: Reject