Learning Audio-Visual Dereverberation

1 INTRODUCTION . Audio reverberation occurs when multiple reflections from surfaces and objects in the environment build up then decay , altering the original audio signal . While reverberation bestows a realistic sense of spatial context , it also can degrade a listener ’ s experience . In particular , the quality of human speech is greatly affected by reverberant environments—as illustrated by how difficult it can be to parse the words of a family member speaking loudly from another room in the house , a tour guide describing the artwork down the hall of a magnificent cavernous cathedral , or a colleague participating in a Zoom call from a cafe . Similarly , automatic speech recognition ( ASR ) suffers when given reverberant speech input ( Kinoshita et al. , 2016 ; Szöke et al. , 2019 ; Watanabe et al. , 2020 ) . Thus there is great need for dereverberation algorithms that can strip away reverb effects for speech enhancement , recognition , and other downstream tasks , which could in turn benefit many applications in teleconferencing , assistive hearing devices , augmented reality , and video indexing . The audio community has made steady progress on speech dereverberation ( Han et al. , 2015 ; Wu et al. , 2016 ; Zhao et al. , 2019b ) . While past approaches have tackled the problem with signal processing and statistical techniques ( Nakatani et al. , 2010 ; Naylor & Gaubitch , 2010 ) , many modern approaches use neural networks to learn a mapping from reverberant to clean spectrograms ( Han et al. , 2015 ; Ernst et al. , 2018 ; Fu et al. , 2019 ) . To our knowledge , all existing models for dereverberation rely purely on audio . This underconstrains the dereverberation task since the latent parameters of the recording space are not discernible from the audio alone . However , we observe that in many practical settings of interest—video conferencing , augmented reality , Web video indexing—reverberant audio is naturally accompanied by a visual ( video ) stream . Importantly , the visual stream offers valuable cues about the room acoustics affecting reverberation : where are the walls , how are they shaped , where is the human speaker , what is the layout of major furniture , what are the room ’ s dominant materials ( which affect absorption , reflection , and refraction ) , and even what is the facial appearance and/or body shape of the person speaking ( since body shape determines the acoustic properties of a person ’ s speech , and reverb time is frequency dependent ) . For example , reverb is typically stronger when the speaker is further away ; speech is more reverberant in a large church or hallway ; heavy carpet absorbs more sound . See Figure 1 . Our idea is to learn to dereverberate speech from audio-visual observations . In this task , the input is reverberant speech and visual observations of the environment surrounding the human speaker , and the output is a prediction of the clean source audio . To tackle this problem , there are two key technical challenges . First , how to model the multi-modal dereverberation process in order to infer the latent clean audio . Second , how to secure appropriate training data spanning a variety of physical environments for which we can sample speech with known ground truth ( non-reverberant , anechoic ) audio . The latter is non-trivial because ordinary audio/video recordings are themselves corrupted by reverberation but lack the ground truth source signal we wish to recover . For the modeling challenge , we introduce an end-to-end approach called Visually-Informed Dereverberation of Audio ( VIDA ) . VIDA consists of a Visual Acoustics Network ( VAN ) that learns reverberation properties of the room geometry , object locations , and speaker position . Coupled with a multi-modal UNet dereverberation module , it learns to remove the reverberations from a single-channel audio stream . In addition , we propose an audio-visual ( AV ) matching loss to enforce consistency between the visually-inferred reverberation features and those inferred from the audio signal . We leverage the outputs of our model for multiple downstream tasks : speech enhancement , speech recognition , and speaker verification . Next , to address the training data challenge , we develop a new large-scale dataset using SoundSpaces ( Chen et al. , 2020 ) , a 3D simulator for real-world scanned environments that allows both visual and acoustic rendering . We insert “ clean '' audio voices together with a 3D humanoid model at various positions within an array of indoor environments , then sample the images and properly reverberating audio when placing the receiver microphone and camera at other positions in the same house . This strategy allows sampling realistic audio-visual instances coupled with ground truth raw audio to train our model , and it has the added benefit of allowing controlled studies that vary the parameters of the capture setting . As we will show , the data also supports sim2real transfer . We first train and evaluate our model on 82 large-scale environments—each a multi-room home containing a variety of objects—coupled with speech samples from the LibriSpeech dataset ( Panayotov et al. , 2015 ) , then further test with real-world data . We consider both near-field and far-field settings where the human speaker is in-view or quite far from the camera , respectively . The proposed model outperforms methods restricted to the audio stream , improves the state of the art for multiple tasks with speech enhancement , and transfers successfully to real-world data . Our main contributions are to 1 ) present the task of audio-visual dereverberation , 2 ) address it with a new multi-modal modeling approach and a novel reverb-visual matching loss , 3 ) provide a benchmark evaluation framework built on both SoundSpaces and real data , and 4 ) demonstrate the utility of AV dereverberation for multiple practical tasks . Overall , our work shows the potential for speech enhancement models to leverage room acoustics by seeing the 3D environment . 2 RELATED WORK . Audio dereverberation and speech enhancement . Audio dereverberation and speech enhancement have a long and rich literature ( Neely & Allen , 1979 ; Miyoshi & Kaneda , 1988 ; Naylor & Gaubitch , 2010 ; Kinoshita et al. , 2016 ) . While dereverberation can be done with microphone arrays , we focus on single audio channel approaches , which require fewer assumptions about the input data . Recent deep learning methods achieve promising results to dereverberate ( Han et al. , 2015 ; Wu et al. , 2016 ; Zhao et al. , 2019b ; Su et al. , 2020b ) , denoise ( Xu et al. , 2019 ; Fu et al. , 2019 ; Su et al. , 2020b ) , or separate ( Hershey et al. , 2016 ; Stoller et al. , 2018 ) the audio stream using audio input alone , which can improve downstream speech recognition ( Ko et al. , 2017 ; Kinoshita et al. , 2016 ; Watanabe et al. , 2020 ) and speaker recognition ( Snyder et al. , 2018 ) . Acoustic simulations can provide data augmentation during training ( Kinoshita et al. , 2016 ; Ko et al. , 2017 ; Zhao et al. , 2020 ) . Some work targets “ room-aware '' deep audio features capturing reverberation properties ( e.g. , RT60 ) ( Giri et al. , 2015 ) , or synthesizes reverberation effects via acoustic matching ( Su et al. , 2020a ) . To our knowledge , the only prior work drawing on the visual stream to infer dereverberated audio is limited to using lip regions on near-field faces to first separate out distractor sounds ( Tan et al. , 2020 ) , and does not model anything about the visual scene for dereverberation purposes . In contrast , our model accounts for the full visual scene , far-field speech sources , and even out-of-view speakers . Our approach is the first to learn visual room acoustics for dereverberation , and it yields state-of-the-art results with direct benefits for multiple downstream tasks . Visual understanding of room acoustics . The room impulse response ( RIR ) is the transfer function capturing the room acoustics for arbitrary source stimuli ; once convolved with a sound waveform , it produces the sound of that source in the context of the particular physical space . While traditionally measured with specialized equipment in the room itself ( Stan et al. , 2002 ; Holters et al. , 2009 ) or else simulated with sound propagation models ( Allen & Berkley , 1979 ; Chen et al. , 2020 ; Murphy et al. , 2007 ) , recent work explores estimating an RIR from an input image using CNNs ( Kon & Koike , 2019 ) or conditional GANs ( Singh et al. , 2021 ) in order to simulate reverberant sound for a given environment . Video-based methods can lift monaural audio into its spatialized ( binaural , ambisonic ) counterpart in order to create an immersive audio experience for human listeners ( Morgado et al. , 2018 ; Gao & Grauman , 2019 ; Li et al. , 2018 ) . Such methods share our interest in learning the visual properties of a scene that influence the audio channel . However , unlike any of the above methods , rather than generate spatialized audio to benefit human listeners in augmented or virtual reality , our goal is to dereverberate audio— removing the effects of the room acoustics—to benefit automatic speech analysis . In addition , prior methods use imagery taken at camera positions at an unknown offset from the microphone , i.e. , conflating all RIRs for a scene with one image , which limits them to a coarse characterization of the environment ( Marco Jeub , 2009 ; Jeub et al. , 2010 ; Murphy & Shelley , 2010 ) . In contrast , our data and model align the camera and microphone to capture novel fine-grained audio-visual properties , including the human speaker ’ s location with respect to the microphone when the speaker is in view . Audio-visual embodied learning in realistic simulations . Recent work in embodied AI explores how vision and sound together can help agents move intelligently in 3D environments . Driven in part by new tools for audio-visual ( AV ) simulations in realistic scanned environments ( Chen et al. , 2020 ) , new research develops deep reinforcement learning approaches to train agents to navigate to sounding objects ( Chen et al. , 2020 ; Gan et al. , 2020 ) , explore unmapped environments ( Dean et al. , 2020 ) , or move around to better separate multiple overlapping sounds in a house ( Majumder et al. , 2021 ) . Our work also leverages state-of-the-art AV simulations for learning , but our objective and models are entirely different . Rather than train virtual robots to move intelligently , our aim is to clean reverberant audio for better speech analysis . Audio-visual learning from video . Multi-modal video understanding has experienced a resurgence of work in the vision , audio , and machine learning literature in recent years . This includes exciting advances in self-supervised cross-modal feature learning from video ( Morgado et al. , 2020 ; Alwassel et al. , 2020 ; Korbar et al. , 2018 ) , localizing objects in video with both sight and sound ( Hu et al. , 2020 ) , and audio-visual speech enhancement or separation ( Ephrat et al. , 2018 ; Owens & Efros , 2018 ; Zhao et al. , 2019a ; Afouras et al. , 2018 ; 2020 ; Michelsanti et al. , 2020 ; Sadeghi et al. , 2020 ; Zhou et al. , 2019 ; Hou et al. , 2017 ) . None of these methods address speech deverberation .

Summary: This paper examines an audio-visual approach for dereverberation, where dereverberation of speech is conditioned on RGB and depth images (either field-of-view or panoramic). It proposes a dataset for this task based on real-world 3D scans of homes, using Librispeech data. Real data is also used The model is based on a U-Net conditioned on embeddings extracted by a "visual acoustics" network doing direct spectrogram prediction. The method is evaluated using PESQ, WER for speech recognition, and EER for speaker verification on synthetic and real data. The audio-visual method is found to perform marginally better than an audio-only version of the model. The audio-only version of the model outperforms several baselines from the literature on synthetic data, with mixed results on real data.