Learning Audio-Visual Dereverberation

1 INTRODUCTION . Audio reverberation occurs when multiple reflections from surfaces and objects in the environment build up then decay , altering the original audio signal . While reverberation bestows a realistic sense of spatial context , it also can degrade a listener ’ s experience . In particular , the quality of human speech is greatly affected by reverberant environments—as illustrated by how difficult it can be to parse the words of a family member speaking loudly from another room in the house , a tour guide describing the artwork down the hall of a magnificent cavernous cathedral , or a colleague participating in a Zoom call from a cafe . Similarly , automatic speech recognition ( ASR ) suffers when given reverberant speech input ( Kinoshita et al. , 2016 ; Szöke et al. , 2019 ; Watanabe et al. , 2020 ) . Thus there is great need for dereverberation algorithms that can strip away reverb effects for speech enhancement , recognition , and other downstream tasks , which could in turn benefit many applications in teleconferencing , assistive hearing devices , augmented reality , and video indexing . The audio community has made steady progress on speech dereverberation ( Han et al. , 2015 ; Wu et al. , 2016 ; Zhao et al. , 2019b ) . While past approaches have tackled the problem with signal processing and statistical techniques ( Nakatani et al. , 2010 ; Naylor & Gaubitch , 2010 ) , many modern approaches use neural networks to learn a mapping from reverberant to clean spectrograms ( Han et al. , 2015 ; Ernst et al. , 2018 ; Fu et al. , 2019 ) . To our knowledge , all existing models for dereverberation rely purely on audio . This underconstrains the dereverberation task since the latent parameters of the recording space are not discernible from the audio alone . However , we observe that in many practical settings of interest—video conferencing , augmented reality , Web video indexing—reverberant audio is naturally accompanied by a visual ( video ) stream . Importantly , the visual stream offers valuable cues about the room acoustics affecting reverberation : where are the walls , how are they shaped , where is the human speaker , what is the layout of major furniture , what are the room ’ s dominant materials ( which affect absorption , reflection , and refraction ) , and even what is the facial appearance and/or body shape of the person speaking ( since body shape determines the acoustic properties of a person ’ s speech , and reverb time is frequency dependent ) . For example , reverb is typically stronger when the speaker is further away ; speech is more reverberant in a large church or hallway ; heavy carpet absorbs more sound . See Figure 1 . Our idea is to learn to dereverberate speech from audio-visual observations . In this task , the input is reverberant speech and visual observations of the environment surrounding the human speaker , and the output is a prediction of the clean source audio . To tackle this problem , there are two key technical challenges . First , how to model the multi-modal dereverberation process in order to infer the latent clean audio . Second , how to secure appropriate training data spanning a variety of physical environments for which we can sample speech with known ground truth ( non-reverberant , anechoic ) audio . The latter is non-trivial because ordinary audio/video recordings are themselves corrupted by reverberation but lack the ground truth source signal we wish to recover . For the modeling challenge , we introduce an end-to-end approach called Visually-Informed Dereverberation of Audio ( VIDA ) . VIDA consists of a Visual Acoustics Network ( VAN ) that learns reverberation properties of the room geometry , object locations , and speaker position . Coupled with a multi-modal UNet dereverberation module , it learns to remove the reverberations from a single-channel audio stream . In addition , we propose an audio-visual ( AV ) matching loss to enforce consistency between the visually-inferred reverberation features and those inferred from the audio signal . We leverage the outputs of our model for multiple downstream tasks : speech enhancement , speech recognition , and speaker verification . Next , to address the training data challenge , we develop a new large-scale dataset using SoundSpaces ( Chen et al. , 2020 ) , a 3D simulator for real-world scanned environments that allows both visual and acoustic rendering . We insert “ clean '' audio voices together with a 3D humanoid model at various positions within an array of indoor environments , then sample the images and properly reverberating audio when placing the receiver microphone and camera at other positions in the same house . This strategy allows sampling realistic audio-visual instances coupled with ground truth raw audio to train our model , and it has the added benefit of allowing controlled studies that vary the parameters of the capture setting . As we will show , the data also supports sim2real transfer . We first train and evaluate our model on 82 large-scale environments—each a multi-room home containing a variety of objects—coupled with speech samples from the LibriSpeech dataset ( Panayotov et al. , 2015 ) , then further test with real-world data . We consider both near-field and far-field settings where the human speaker is in-view or quite far from the camera , respectively . The proposed model outperforms methods restricted to the audio stream , improves the state of the art for multiple tasks with speech enhancement , and transfers successfully to real-world data . Our main contributions are to 1 ) present the task of audio-visual dereverberation , 2 ) address it with a new multi-modal modeling approach and a novel reverb-visual matching loss , 3 ) provide a benchmark evaluation framework built on both SoundSpaces and real data , and 4 ) demonstrate the utility of AV dereverberation for multiple practical tasks . Overall , our work shows the potential for speech enhancement models to leverage room acoustics by seeing the 3D environment . 2 RELATED WORK . Audio dereverberation and speech enhancement . Audio dereverberation and speech enhancement have a long and rich literature ( Neely & Allen , 1979 ; Miyoshi & Kaneda , 1988 ; Naylor & Gaubitch , 2010 ; Kinoshita et al. , 2016 ) . While dereverberation can be done with microphone arrays , we focus on single audio channel approaches , which require fewer assumptions about the input data . Recent deep learning methods achieve promising results to dereverberate ( Han et al. , 2015 ; Wu et al. , 2016 ; Zhao et al. , 2019b ; Su et al. , 2020b ) , denoise ( Xu et al. , 2019 ; Fu et al. , 2019 ; Su et al. , 2020b ) , or separate ( Hershey et al. , 2016 ; Stoller et al. , 2018 ) the audio stream using audio input alone , which can improve downstream speech recognition ( Ko et al. , 2017 ; Kinoshita et al. , 2016 ; Watanabe et al. , 2020 ) and speaker recognition ( Snyder et al. , 2018 ) . Acoustic simulations can provide data augmentation during training ( Kinoshita et al. , 2016 ; Ko et al. , 2017 ; Zhao et al. , 2020 ) . Some work targets “ room-aware '' deep audio features capturing reverberation properties ( e.g. , RT60 ) ( Giri et al. , 2015 ) , or synthesizes reverberation effects via acoustic matching ( Su et al. , 2020a ) . To our knowledge , the only prior work drawing on the visual stream to infer dereverberated audio is limited to using lip regions on near-field faces to first separate out distractor sounds ( Tan et al. , 2020 ) , and does not model anything about the visual scene for dereverberation purposes . In contrast , our model accounts for the full visual scene , far-field speech sources , and even out-of-view speakers . Our approach is the first to learn visual room acoustics for dereverberation , and it yields state-of-the-art results with direct benefits for multiple downstream tasks . Visual understanding of room acoustics . The room impulse response ( RIR ) is the transfer function capturing the room acoustics for arbitrary source stimuli ; once convolved with a sound waveform , it produces the sound of that source in the context of the particular physical space . While traditionally measured with specialized equipment in the room itself ( Stan et al. , 2002 ; Holters et al. , 2009 ) or else simulated with sound propagation models ( Allen & Berkley , 1979 ; Chen et al. , 2020 ; Murphy et al. , 2007 ) , recent work explores estimating an RIR from an input image using CNNs ( Kon & Koike , 2019 ) or conditional GANs ( Singh et al. , 2021 ) in order to simulate reverberant sound for a given environment . Video-based methods can lift monaural audio into its spatialized ( binaural , ambisonic ) counterpart in order to create an immersive audio experience for human listeners ( Morgado et al. , 2018 ; Gao & Grauman , 2019 ; Li et al. , 2018 ) . Such methods share our interest in learning the visual properties of a scene that influence the audio channel . However , unlike any of the above methods , rather than generate spatialized audio to benefit human listeners in augmented or virtual reality , our goal is to dereverberate audio— removing the effects of the room acoustics—to benefit automatic speech analysis . In addition , prior methods use imagery taken at camera positions at an unknown offset from the microphone , i.e. , conflating all RIRs for a scene with one image , which limits them to a coarse characterization of the environment ( Marco Jeub , 2009 ; Jeub et al. , 2010 ; Murphy & Shelley , 2010 ) . In contrast , our data and model align the camera and microphone to capture novel fine-grained audio-visual properties , including the human speaker ’ s location with respect to the microphone when the speaker is in view . Audio-visual embodied learning in realistic simulations . Recent work in embodied AI explores how vision and sound together can help agents move intelligently in 3D environments . Driven in part by new tools for audio-visual ( AV ) simulations in realistic scanned environments ( Chen et al. , 2020 ) , new research develops deep reinforcement learning approaches to train agents to navigate to sounding objects ( Chen et al. , 2020 ; Gan et al. , 2020 ) , explore unmapped environments ( Dean et al. , 2020 ) , or move around to better separate multiple overlapping sounds in a house ( Majumder et al. , 2021 ) . Our work also leverages state-of-the-art AV simulations for learning , but our objective and models are entirely different . Rather than train virtual robots to move intelligently , our aim is to clean reverberant audio for better speech analysis . Audio-visual learning from video . Multi-modal video understanding has experienced a resurgence of work in the vision , audio , and machine learning literature in recent years . This includes exciting advances in self-supervised cross-modal feature learning from video ( Morgado et al. , 2020 ; Alwassel et al. , 2020 ; Korbar et al. , 2018 ) , localizing objects in video with both sight and sound ( Hu et al. , 2020 ) , and audio-visual speech enhancement or separation ( Ephrat et al. , 2018 ; Owens & Efros , 2018 ; Zhao et al. , 2019a ; Afouras et al. , 2018 ; 2020 ; Michelsanti et al. , 2020 ; Sadeghi et al. , 2020 ; Zhou et al. , 2019 ; Hou et al. , 2017 ) . None of these methods address speech deverberation .

Summary: This paper presents a deep neural net-based dereverberation algorithm that uses both audio and video modalities. Based on the observation that a visual scene captured by a camera conveys information that is related to room characteristics, the authors propose a visually informed audio dereverberation method that aims to extract clean, anechoic speech from reverberant speech. In doing so, they first construct a large audio-visual dataset synthesized using a 3D simulator for real-world scanned environments and LibriSpeech data. They then train deep neural networks that take as inputs both visual data (RGB and depth images) and audio data (reverberant speech), and output clean speech. When training, two types of losses - one for clean speech spectrogram estimation and the other for reverb-visual matching - are used. Through the experiments with several downstream tasks for speech, they showed that the proposed audio-visual dererberation method outperforms the baseline models, both for synthetic and real-world test data.