Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds

1 INTRODUCTION . Audio-visual machine perception has been undergoing a renaissance in recent years driven by advances in large-scale deep learning . A motivating observation is the interplay in human perception between auditory and visual perception . We understand the world by parsing it into the objects that are the sources of the audio and visual signals we can perceive . However , the sounds and sights produced by these sources have rather different and complementary properties . Objects may make sounds intermittently , whereas their visual appearance is typically persistent . The visual percepts of different objects tend to be spatially distinct , whereas sounds from different sources can blend together and overlap in a single signal , making it difficult to separately perceive the individual sources . ∗Work done during an internship at Google . This suggests that there is something to be gained by aligning our audio and visual percepts : if we can identify which audio signals correspond to which visual objects , we can selectively attend to an object ’ s audio signal by visually selecting the object . This intuition motivates using vision as an interface for audio processing , where a primary problem is to selectively preserve desired sounds , while removing unwanted sounds . In some tasks , such as speech enhancement , the desired sounds can be selected by their class : speech versus non-speech in this case . In an open-domain setting , the selection of desired sounds is at the user ’ s discretion . This presents a user-interface problem : it is challenging to select sources in an efficient way using audio . This problem can be greatly simplified in the audio-visual case if we use video selection as a proxy for audio selection , for example , by selecting sounds from on-screen objects , and removing off-screen sounds . Recent work has used video for selection and separation of speech ( Ephrat et al. , 2018 ; Afouras et al. , 2020 ) or music ( Zhao et al. , 2018 ; Gao & Grauman , 2019 ; Gan et al. , 2020 ) . However , systems that address this for arbitrary sounds ( Gao et al. , 2018 ; Rouditchenko et al. , 2019 ; Owens & Efros , 2018 ) may be useful in more general cases , such as video recording , where the sounds of interest can not be defined in advance . The problem of associating arbitrary sounds with their visual objects is challenging in an open domain . Several complications arise that have not been fully addressed by previous work . First , a large amount of training data is needed in order to cover the space of possible sound . Supervised methods require labeled examples where isolated on-screen sounds are known . The resulting data collection and labeling burden limits the amount and quality of available data . To overcome this , we propose an unsupervised approach using mixture invariant training ( MixIT ) ( Wisdom et al. , 2020 ) , that can learn to separate individual sources from in-the-wild videos , where the on-screen and off-screen sounds are unknown . Another problem is that different audio sources may correspond to a dynamic set of on-screen objects in arbitrary spatial locations . We accommodate this by using attention mechanisms that align each hypothesized audio source with the different spatial and temporal positions of the corresponding objects in the video . Finally we need to determine which audio sources appear on screen , in the absence of strong labels . This is handled using a weakly trained classifier for sources based on audio and video embeddings produced by the attention mechanism . 2 RELATION TO PREVIOUS WORK . Separation of arbitrary sounds from a mixture , known as “ universal sound separation , ” was recently shown to be possible with a fixed number of sounds ( Kavalerov et al. , 2019 ) . Conditional information about which sound classes are present can improve separation performance ( Tzinis et al. , 2020 ) . The FUSS dataset ( Wisdom et al. , 2021 ) expanded the scope to separate a variable number of sounds , in order to handle more realistic data . A framework has also been proposed where specific sound classes can be extracted from input sound mixtures ( Ochiai et al. , 2020 ) . These approaches require curated data containing isolated sounds for training , which prevents their application to truly open-domain data and introduces difficulties such as annotation cost , accurate simulation of realistic acoustic mixtures , and biased datasets . To avoid these issues , a number of recent works have proposed replacing the strong supervision of reference source signals with weak supervision labels from related modalities such as sound class ( Pishdadian et al. , 2020 ; Kong et al. , 2020 ) , visual input ( Gao & Grauman , 2019 ) , or spatial location from multi-microphone recordings ( Tzinis et al. , 2019 ; Seetharaman et al. , 2019 ; Drude et al. , 2019 ) . Most recently , Wisdom et al . ( 2020 ) proposed mixture invariant training ( MixIT ) , which provides a purely unsupervised source separation framework for a variable number of latent sources . A variety of research has laid the groundwork towards solving audio-visual on-screen source separation ( Michelsanti et al. , 2020 ) . Generally , the two main approaches are to use audio-visual localization ( Hershey & Movellan , 2000 ; Senocak et al. , 2018 ; Wu et al. , 2019 ; Afouras et al. , 2020 ) , or object detection networks , either supervised ( Ephrat et al. , 2018 ; Gao & Grauman , 2019 ; Gan et al. , 2020 ) or unsupervised ( Zhao et al. , 2018 ) , to predict visual conditioning information . However , these works only consider restricted domains such as speech ( Hershey & Casey , 2002 ; Ephrat et al. , 2018 ; Afouras et al. , 2020 ) or music ( Zhao et al. , 2018 ; Gao & Grauman , 2019 ; Gan et al. , 2020 ) . Gao et al . ( 2018 ) reported results with videos from a wide domain , but relied on supervised visual object detectors , which precludes learning about the appearance of sound sources outside of a closed set of classes defined by the detectors . Rouditchenko et al . ( 2019 ) proposed a system for a wide domain of sounds , but required sound class labels as well as isolated sounds from these classes . Our approach avoids the supervision of class labels and isolated sources in order to handle unknown visual and sound classes occurring in multi-source data . Towards learning directly from a less restrictive open domain of in-the-wild video data , Tian et al . ( 2018 ) learned to localize audio-visual events in unconstrained videos and presented an ad hoc dataset . Korbar et al . ( 2018 ) pretrained models to discern temporal synchronization of audio-video pairs , and demonstrated promising results on action recognition and audio classification . Arandjelovic & Zisserman ( 2017 ) took a similar approach by classifying audio-visual correspondences of pairs of one video frame and one second of audio . Hu et al . ( 2020 ) proposed a curriculum learning approach where the model gradually learns harder examples to separate . Closest to our work is the approach of Owens & Efros ( 2018 ) , a self-supervised audio-visual onscreen speech separation system based on temporal audio-visual alignment . However , Owens & Efros ( 2018 ) assumes training videos containing only on-screen sources , and it is unclear how to adapt it to the case where training videos include off-screen sources . Our approach significantly differs from these prior works in that we do not restrict our domain to musical instruments or human speakers , and we train and test with real in-the-wild videos containing an arbitrary number of objects with no object class restrictions . Our proposed framework can deal with noisy labels ( e.g . videos with no on-screen sounds ) , operate on a completely open-domain of in-the-wild videos , and effectively isolate sounds coming from on-screen objects . We address the following task , which extends the formulation of the on-screen speech separation problem ( Owens & Efros , 2018 ) . Given an input video , the goal is to separate all sources that constitute the input mixture , and then estimate an audio-visual correspondence score for each separated source . These probability scores should be high for separated sources which are apparent on-screen , and low otherwise . The separated audio sources , weighted by their estimated on-screen probabilities , can be summed together to reconstruct the on-screen mixture . We emphasize that our approach is more generally applicable than previous proposals , because real-world videos may contain an unknown number of both on-screen and off-screen sources belonging to an undefined ontology of classes . We make the following contributions in this paper : 1 . We provide the first solution for training an unsupervised , open-domain , audio-visual onscreen separation system from scratch on real in-the-wild video data , with no requirement on modules such as object detectors that require supervised data . 2 . We develop a new dataset for the on-screen audio-visual separation task , drawn from 2,500 hours of unlabeled videos from YFCC100m , and 55 hours of videos that are human-labeled for presence of on-screen and off-screen sounds . 3 MODEL ARCHITECTURE . The overall architecture of AudioScope is built from the following blocks : an image embedding network , an audio separation network , an audio embedding network , an audio-visual attention mechanism , and an on-screen classifier ( see Figure 2 ) . The separation and embedding networks are based on prior work and are described in the following subsections . However , the main focus of this work is the overall architecture , as well as the training framework and loss functions . The video is analyzed with the image embedding network , which generates local embeddings for each of 64 locations within each frame , as well as an embedding of the whole frame . These embeddings are used both as a conditioning input to an audio separation network , as well as an input for classification of the on-screen sounds . The audio separation network takes the mixed input waveform as input , and generates a fixed number of output waveforms , a variable number of which are non-zero depending on the estimated number of sources in the mixture . Conditioning on the video enables the separation to take advantage of cues about the sources present when performing separation . The audio embedding network is applied to each estimated source to obtain one embedding per frame for each source . These audio embeddings are then pooled over time and used in the audio-visual spatio-temporal attention network to retrieve , for each source , a representation of the visual activity that best matches the audio , similar to the associative maps extracted from the internal network representations proposed by Harwath et al . ( 2018 ) . The architecture is designed to address the problem of unsupervised learning on in-the-wild opendomain data . First , because the target training videos can contain both on-screen and off-screen sounds , training a system to directly produce the audio of the target video would encourage inclusion of off-screen sounds as well as on-screen ones1 . Our proposed multi-source separation network instead produces latent source estimates using an unsupervised MixIT objective , which has been shown to perform well at general sound separation ( Wisdom et al. , 2020 ) . By decoupling separation from on-screen classification , our architecture facilitates the use of robust objectives that allow some of the sources to be considered off-screen , even if they appear in the soundtrack of the target videos . The audio-visual attention architecture is motivated by the alignment problem between audio and video : sound source objects in video may be localized , may move over time , and may be present before and after the corresponding audio activity . Because of the open domain we can not rely on a pre-defined set of object detectors to anchor the video representations of on-screen sources , as is done in some prior works ( Ephrat et al. , 2018 ; Gao & Grauman , 2019 ; Gan et al. , 2020 ) . Instead we propose attention to find the video representations that correspond to a source in a more flexible way . The proposed strategy of temporal pooling of the audio embeddings , before using them in the spatiotemporal attention , allows the network to derive embeddings that represent the active segments of the source audio , and ignore the ambiguous silent regions . In the present model , video is analyzed at a low frame rate , and so the audio-visual correspondence is likely based on relatively static properties of the objects , rather than the synchrony of their motion with the audio . In this case , a single time-invariant representation of the audio may be sufficient as a proof of concept . However , in future work , with higher video frame rates , it may be worthwhile to consider using attention to align sequences of audio and video embeddings in order to detect synchrony in their activity patterns . The on-screen classifier operates on an audio embedding for one estimated source , as well as the video embedding retrieved by the spatio-temporal attention mechanism , using a dense network . This presumably allows detection of the congruence between the embeddings . To provide additional context for this decision , a global video embedding , produced by temporal pooling , is provided as an additional input . Many alternative choices are possible for this classifier design , which we leave for future work , such as using a more complex classification architecture , or providing additional audio embeddings as input . 1We train such a system in Appendix A.3.5 , and find that it is not an effective approach .

Summary: This paper describes a system for separating "on-screen" sounds from "off-screen" sounds in an audio-visual task, meaning sounds that are associated with objects that are visible in a video versus not. It is trained to do this using mixture invariant training to separate synthetic mixtures of mixtures. It is evaluated on a subset of the YFCC100m that is annotated by human raters as to whether the clips have on-screen, off-screen, or both types of sounds, with the predictions of a previously described model  (Jansen et al, 2020) helping to reduce the number with only off-screen sounds. The predictions are evaluated in terms of how well they can estimate the true on-screen sound (in terms of SI-SNR) and how well they can reject off-screen sound (in terms of a metric called off-screen suppression ratio, OSR). The results show that the system can successfully distinguish between on- and off-screen sound, but that different training regimens lead to different tradeoffs in these two metrics. The system with the best SI-SNR (8.0 dB) is trained using just data from the previous model along with the mixture invariant training criterion.