Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds
Authors: Efthymios Tzinis
Scott Wisdom
Aren Jansen
Shawn Hershey
Tal Remez
Dan Ellis
John R. Hershey
Venue: ICLR 2021 Poster (2021)
Keywords: Audio-visual sound separation
in-the-wild data
unsupervised learning
self-supervised learning
universal sound separation

Abstract: Recent progress in deep learning has enabled many advances in sound separation and visual scene understanding. However, extracting sound sources which are apparent in natural videos remains an open problem. In this work, we present AudioScope, a novel audio-visual sound separation framework that can be trained without supervision to isolate on-screen sound sources from real in-the-wild videos. Prior audio-visual separation work assumed artificial limitations on the domain of sound classes (e.g., to speech or music), constrained the number of sources, and required strong sound separation or visual segmentation labels. AudioScope overcomes these limitations, operating on an open domain of sounds, with variable numbers of sources, and without labels or prior visual segmentation.  The training procedure for AudioScope uses mixture invariant training (MixIT) to separate synthetic mixtures of mixtures (MoMs) into individual sources, where noisy labels for mixtures are provided by an unsupervised audio-visual coincidence model. Using the noisy labels, along with attention between video and audio features, AudioScope learns to identify audio-visual similarity and to suppress off-screen sounds. We demonstrate the effectiveness of our approach using a dataset of video clips extracted from open-domain YFCC100m video data. This dataset contains a wide diversity of sound classes recorded in unconstrained conditions, making the application of previous methods unsuitable. For evaluation and semi-supervised experiments, we collected human labels for presence of on-screen and off-screen sounds on a small subset of clips.

Published as a conference paper at ICLR 2021
INTO THE WILD WITH AUDIOSCOPE: UNSUPERVISED AUDIO- VISUAL SEPARATION OF ON-SCREEN SOUNDS
Efthymios Tzinis" Scott Wisdom² Aren Jansen?, Shawn Hershey?, Tal Remez",
Daniel P.W. Elliss, John R. Hershey UUniversity Illinois Urbana- Champpana etzinis24 iilinis2e 1 1illing wiiinii.eeee edu
?Google Research
scot
ABSTRACT
Recent progress in deep learning has enabled many advances in sound separation and visual scene understanding. However, extracting sound sources which are apparent in natural videos remains an open problem. In this work, we present AudioScope, novel audio-visual sound separation framework that can trained without supervision to isolate on-screen sound sources from real in-the-wild videos. Prior audio-visual separation work assumed artificial limitations on the domain of sound classes (e.g., to speech music), constrained the number of sources, and required strong sound separation or visual segmentation labels. AudioScope overcomes these limitations, operating an open domain of sounds, with variable numbers of sources, and without labels or prior visual segmentation. The training mixture invariant training (MixITT) separate
procedure for AudioScope synthetic mixtures of mixtures (MoMs) into individual sources, where noisy labels for mixtures are provided by an unsupervised audio-visual coincidence model. Using the noisy labels, along with attention between video and audio features, AudioScope learns to identify audio-visual similarity and to suppress off-screen sounds. We demonstrate the effectiveness of our approach using a dataset of video clips extracted from open-domain YFCC100m video data. This dataset contains a wide diversity of sound classes recorded in unconstrained conditions, making the application of previous methods unsuitable. For evaluation and semi-supervised experiments, collected human labels for presence of on-screen and off-screen sounds on small subset of clips.
INTRODUCTION
Video frame
On-screen audic
Off audio
audio
Attention map
On-screen estimate
Figure 1: AudioScope separating on-screen bird chirping from wind noise and off-screen sounds from fireworks and human laugh. More demos online at httpts::///uut https://audioscope github. io.
Audio-visuaa machine perception has been undergoing a renaissance in recent years driven by advances in large-scale deep learning. motivating observation the interplay human perception between auditory and visual perception. We understand the world by parsing it into the objects that are the sources of the audio and visual signals we can perceive. However, the sounds and sights produced by these sources have rather different and complementary properties. Objects may make sounds intermittently, whereas their visual appearance is typically persistent. The visual percepts of different objects tend to be spatially distinct, whereas sounds from different sources blend together and overlap in a single signal, making difficult to separately perceive the individual sources.
Work done during intersship Gongle.

---

Published as a conference paper at ICLR 2021
This suggests that there is something to be gained by aligning our audio and visual percepts: if we can identify which audio signals correspond to which visual objects, we can selectively attend to an object's audio signal by visually selecting the object.
This intuition motivates using vision as an interface for audio processing, where a primary problem is to selectively preserve desired sounds, while removing unwanted sounds. In some tasks, such as speech enhancement, the desired sounds can be selected by their class: speech versus non-speech in this case. In an open-domain setting, the selection of desired sounds is the user's discretion. This presents a user-interface problem: it is challenging to select sources in an efficient way using audio. This problem can be greatly simplified in the audio-visual case if we use video selection as a proxy for audio selection, for example, by selecting sounds from on-screen objects, and removing off-screen sounds. Recent work has used video for selection and separation of speech (Ephrat et al. 2018, Afouras A et al., 2020) or music (Zhao et al. 2018 Gao & Grauman, 2019: Gan et al.) (020) However, systems that address this for arbitrary sounds (Gao et al. [0188] Rouditchenko et al. 2019)
Owens & Efros 2018) may be useful in more general cases, such video recording, where the sounds of interest cannot be defined in advance.
The problem of associating arbitrary sounds with their visual objects is challenging in an open domain. Several complications arise that have not been fully addressed by previous work. First, a large amount of training data is needed in order to cover the space of possible sound. Supervised methods require labeled examples where isolated on-screen sounds are known. The resulting data collection and labeling burden limits the amount and quality of available data. To overcome this, propose an unsupervised approach using mixture invariant training (MixIT) (Wisdom et al. 2020), that can learn to separate individual sources from in-the-wild videos, where the on-screen and off-screen sounds are unknown. Another problem is that different audio sources may correspond to dynamic set of on-screen objects arbitrary spatial locations. We accommodate this by using attention mechanisms that align each hypothesized audio source with the different spatial and temporal positions of the corresponding objects the video. Finally we need to determine which audio sources appear on screen, in the absence of strong labels. This is handled using weakly trained classifier for sources based on audio and video embeddings produced by the attention mechanism.
2 RELATION TO PREVIOUS WORK
Separation of arbitrary sounds from a mixture, known as "universal sound separation," " was recently shown to be possible with fixed number of sounds (Kavalerov et al. (019). Conditional information about which sound classes are present can improve separation performance (Txinis et al.. 20200. The FUSS dataset (Wisdom et al. 202T) expanded the scope to separate a variable number of sounds, in order to handle more realistic data. A framework has also been proposed where specific sound classes can be extracted from input sound mixtures (Ochiai et al. 2020). These approaches require curated data containing isolated sounds for training. which prevents their application to truly open-domain data and introduces difficulties such as annotation cost, accurate simulation of realistic acoustic mixtures, and biased datasets.
To avoid these issues, number of recent works have proposed replacing the strong supervision of reference source signals with weak supervision labels from related modalities such as sound class
(Pishdadian al. 2020 Kong et al. 2020)), visual input (Gao & Grauman 2019). or spatial location
from multi--iirophone recordings Tzinis et al. 2019]] (Seetharaman et al., 2019 Drude et al. 2019).
Most recently, WWisdom al. (2020) proposed mixture invariant training (MixlT), ), which provides
purely unsupervised source separation framework for variable number of latent sources.
variety of research has laid the groundwork towards solving audio-visue on-screen source separa- tion (Michelsanti et al. 2020). Generally, the two main approaches are to use
localization
(Hershey & Movellan, 2000, [Snocak et al. 2018 Wu et al. 2019 Afouras et al. 2020). or object
detection networks, either supervised (Ephrat et al. 2018: Gao & Grauman 2019 Gan et al.) (020)
or unsupervised (Zhao et al. (2018), to predict visual conditioning information. However, these works only consider restricted domains such as speech (Hershey & Casey|
Ephrat et al. 2018 Afouras et al. 2020) or music (Zhao et al.; [2018) Gao & Grauman| 2019; Gan et 2020). [ao et al. (2018) reported results with videos from wide domain, but relied on supervised visual object detectors, which precludes learning about the appearance of sound sources outside of a closed set of classes defined by the detectors, Rouditchenko et al. (2019) proposed system for wide domain of sounds,
2

---

Published as a conference paper at ICLR 2021
but required sound class labels as well as isolated sounds from these classes. Our approach avoids the supervision of class labels and isolated sources in order to handle unknown visual and sound classes occurring in multi-source data.
Towards learning directly from a less restrictive open domain of in-the-will video data, Tian et al. (2018) learned to localize audio-visual events in unconstrained videos and presented ad hoc dataset. Korbar et al. (0118) pretrained models to discern temporal synchronization of audio-video pairs, and demonstrated promising results on action recognition and audio classification. Arandjelovic & Zisserman (017)) took a similar approach by classifying audio-visual correspondences of pairs of one video frame and one second of audio. Hu et al. (020) proposed a curriculum learning approach where the model gradually learns harder examples to separate.
Closest our work the approach Owens Efros (2018), a self-supervised audio-visual on- screen speech separation system based on temporal audio-visual alignment. However, Owens & Efros 2018) assumes training videos containing only on-screen sources, and it is unclear how to adapt it to the case where training videos include off-screen sources.
Our approach significantly differs from these prior works in that we do not restrict our domain to musical instruments or human speakers, and we train and test with real in-the-wild videos containing an arbitrary number of objects with no object class restrictions. Our proposed framework can deal with noisy labels (e.g. videos with on-screen sounds), operate completely open-domain of in-the-wild wild videos, and effectively isolate sounds coming from on-screen objects.
We address the following task, which extends the formulation of the on-screen speech separation problem (Owens & Efros 2018). Given an input video, the goal is to separate all sources that constitute the input mixture, and then estimate an visual correspondence score for each separated source. These probability scores should be high for separated sources which are apparent on-screen, and low otherwise. The separated audio sources, weighted by their estimated on-screen probabilities, can be summed together to reconstruct the on-screen mixture. We emphasize that our approach is more generally applicable than previous proposals, because real-world videos may contain an unknown number of both on-screen and off-screen sources belonging to an undefined ontology of classes.
We make the following contributions in this paper:
We provide the first solution for training an unsupervised, open-domain, audio-visual on- screen separation system from scratch on real till video data, with no requirement on modules such as object detectors that require supervised data.
2. We develop new dataset for the on-screen audio-visual separation task, drawn from 2,500 hours of unlabeled videos from YFCC100m, and 55 hours of videos that are human-labeldd for presence of on-screen and off-screen sounds.
MODEL ARCHITECTURE
The overall architecture of AudioScope is built from the following blocks: an image embedding network, an audio separation network, an audio embedding network, an audio-visual attention mechanism, and an on-screen classifier (see Figure 2 The separation and embedding networks are based on prior work and are described in the following subsections. However, the main focus of this work is the overall architecture, as well as the training framework and loss functions.
The video is analyzed with the image embedding network, which generates local embeddings for each of 64 locations within each frame, as well as an embedding of the whole frame. These embeddings are used both as conditioning input to an audio separation network, as well as an input for classification of the on-screen sounds. The audio separation network takes the mixed input waveform as input, and generates a fixed number of output waveforms, a variable number which are non-zero depending on the estimated number of sources in the mixture. Conditioning on the video enables the separation to take advantage of cues about the sources present when performing separation. The audio embedding network is applied to each estimated source to obtain one embedding per frame for each source. These audio embeddings are then pooled over time and used in the audio-visual apatio-temporal attention network retrieve, for each source, representation of the visual activity that best matches the
3

---

Published as a conference paper at ICLR 2021
audio, similar to the associative maps extracted from the internal network representations proposed by Harwath et al. (2018).
The architecture is designed address the problem of unsupervised learning in-the-wild open- domain data. First, because the target training videos can contain both on-screen and off-screen sounds, training system directly produce the audio of the target video would encourage inclusion of off-screen sounds as well as on-screen one Our proposed multi-source separation network instead produces latent source estimates using an unsupervised MixIT objective, which has been shown to perform well at general sound separation (Wisdom et al. 2020). By decoupling separation from on-screen classification, our architecture facilitates the use of robust objectives that allow some of the sources to be considered off-screen, even if they appear in the soundtrack of the target videos.
The audio-visual attention architecture is motivated by the alignment problem between audio and video: sound source objects in video may be localized, may move over time, and may be present before and after the corresponding audio activity. Because of the open domain we cannot rely on pre-defined set of object detectors to anchor the video representations of on-screen sources, as is done some prior works (Ephrat et al., 2018 8 Gao Grauman, 2019 Gan al. 2020). Instead we propose attention to find the video representations that correspond to source in more flexible way.
The proposed strategy of temporal pooling of the audio embeddings, before using them the spatio- temporal attention, allows the network to derive embeddings that represent the active segments of the source audio, and ignore the ambiguous silent regions. In the present model, vidco is analyzed at low frame rate, and the audio-visual correspondence is likely based relatively static properties of the objects, rather than the synchrony representation of the audio may be sufficient as proof of concept. However, in future work, with higher video frame rates, it may be worthwhile to consider using attention to align sequences of audio and video embeddings in order detect synchrony their activity patterns.
their motion with the audio. In this case, a single time-invariant
The on-screen classifier operates on an audio embedding for one estimated source, as well as the video embedding retrieved by the spatio-temporal attention mechanism, using a dense network. This presumably allows detection of the congruence between the embeddings. To provide additional context for this decision, global video embedding, produced by temporal pooling, is provided as an additional input. Many alternative choices are possible for this classifier design, which we leave for future work, such as using a more complex classification architecture, or providing additional audio embeddings as input.
Image embedding network M
Zy
On-screen classifier
Global video embedding
1 m
Buper meoruua
2V
Spatiotemporal attention
Local spatiotemporal video embedding
2 3 Predictions
S Temporal pooling
Video frames
Classification loss
Video soundtrack
Visual conditioning
m,1:Fi
Pseudo-labels
Audio embedding network M
Audio separation network M
Mixture of mixtures (MoM)
2 4
m
1, 3,4
y
MixIT assignment
Random soundtrack
Estimated sources
SNR loss
Figure 2: AudioScope system diagram with 2 input mixtures and 4 output sources.
We train such system in Appendix A.3.5 and find that it is not effective approach.
4

---

Published as a conference paper at ICLR 2021
3.1 AUDIO SEPARATION NETWORK
The separation network M architecture consists of learnable convolutional encoder and decoder layers with an improved time-domain convolutional network (TDCN++) masking network (Wisdom et al. 2020). A mixture consistency projection (Wisdom al.] 2019) is applied to constrain separated sources to add up to the input mixture. The separation network processes a T-sample input mixture waveform and outputs M estimated sources & E IR IR Internally, the network estimates M masks which are multiplied with the activations of the encoded input mixture. The timeddomain signals 8 are computed by applying the decoder, a transposed convolutional layer, the masked coefficients.
3.2 AUDIO EMBEDDING NETWORK
For each separated source s.. we extract corresponding global audio embedding using the Mo- bileNet v1 architecture (Howard et al. (017) which consists of stacked 2D separable dilated con- volutional blocks with a dense layer at the end. This network M first computes log Mel-scale spectrograms with Fa audio frames from the time-domain separated sources, and then applies stacks of depthwise separable convolutions to produce the Fa x N embedding matrix Z which contains N-dimensional row embedding for each frame. An attentional pooling operation GGirdhar& & Ramanan (017) is used, for each source, m, to form a static audio embedding vec- for & attend[ 7e Zá Z where the average embedding Za
Σ; Zm. is the query vector for source m. The attention mechanism (Bahdanau et al. 2013 is defined follows:
attend(g, K, = a" (v(),, ==sotmaax(tanb (Jk(K)) tanh (fa(o))))
(1)
with query row vector q, the attention weight distribution column vector a, key matrix K, value matrix V, and trainable row-wise dense layers fąp fv, fK, all having conforming dimensions.
3.3 IMAGE EMBEDDING NETWORK
To extract visual features from video frames, again MobileNet v1 architecture. This visual embedding model M is applied independently to each one of the F, input video frames and a static-length embedding is extracted for each image 27, F.....,
Conditioning separation network with the temporal video embedding: The embeddings of the video input Zý can be used to condition the separation network (Tzinis et al. 2020). Specifically, the image embeddings are fed through a dense layer, and a simple nearest neighbor upsampling matches the time dimension to the time dimension of the intermediate separation network activations. These upsampled and transformed image embeddings are concatenated with the intermediate TDCN+ activations and fed as input the separation network layers.
Global video embedding: A global embedding of the video input is extracted using attentional pooling over all video frames, given by z attend( Z", ZY, 2"), where the average embedding Z" E, Zj is the query vector.
Local spatio-temporal video embedding: We also use local features extracted from an intermediate level in the visual convolutional network, that has 8 spatial locations. These are denoted Z where k = (j, n)) indexes video frame j and spatial location index n. These provide spatial features for identification of sources with visual objects be used with audio-visual spatio-temporal attention.
3.4 AUDIO- TEMPORAL ATTENTION
important aspect of this work is to combine audio and visual information in order to infer correspondence between each separated source and the relevant objects in video. This in turn will be used to identify which sources are visible on-screen. To this end, we employ an audio-visual spatio- temporal attention scheme by letting the network attend to the local features of the visual embeddings for each separated source. In this mechanism, we use the audio embedding 2n, as the query input for source and the key and value inputs given by the netio-ttempoal video embeddings, 2v,. As a result, the flattened version of the output source, is av attend( = 7 7.1..
embedding, corresponding to the m-th
5

---

Published as a conference paper at ICLR 2021
3.5 ON-SCREEN CLASSIFIER
To infer the visual presence each separated source, we concatenate the global video embedding z²8, the global audio embedding for each source zm and the corresponding local spatio-temporal audio-visual embedding 2m The concatenated vector is fed through a dense layer fc with logistic activation: In. logistic (fe ([2 2m 2m]
3.6 SEPARATION Loss
We use MixIT separation loss (Wisdom al.] 2020), which optimizes the assignment of M estimated sources å = MM (x1 + 22) to two reference mixtures T1, as follows:
Seep , 22, 8) min (CENR (æ1, [A$]] + LSNK (22, (A8)2))
where the mixing matrix A E 22xM constrained to the set of 2 x M binary matrices where each each source 8m can only be assigned one reference
column sums Due to the constraints mixture. The SNR loss for an estimated signal E R² and target signal t E R' is defined as:
(sNR(1,) 1) 10 (I1 2ll² + 10 *1411))
3.7 CLASSIFICATION Loss
To train the on-screen classifier, we consider the following classification losses. These losses use binary labels ym where given for supervised examples, and in the unsupervised case ymm A
for each source m, where A is the optimial mixing matrix found by the minimization in (2).
We
also the notation R. (mly Um 1, (1,-.., M)} to denote the set of positive labels.
Exact binary cross entropy:
M
Lexact (v. 0) =
Σ
Um log (0m) + (Um log (1 - (
(4)
Multiple-instamee cross entropy: Since some separated sources assigned to the on-screen mixture not on-screen, multiple-instance (MI) (Maron & Lozano-Péres 1998) loss, which minimizes over the set of positive labels R may be more robust:
LMI (y. 0) = min (-log (0m) mER.
Σ
log (1 - Imr))
(5)
m
Active combinations entropy: An alternative to the MI loss, active combinations (AC), corresponds to the minimum loss over all settings P21 (R) of the labels s.t. at least one label is 1:
LAC (v.0)
min se (R)
(- log (9m)
Σ Sg
log (1 Bmm))
(6)
mES
where Φ21 (R) denotes the power set of indices with label of 1.
4 EXPERIMENTAL FRAMEWORK
4.1 DATA PREPARATION
In order to train on real-world audio-visual recording enviromments for our open-domain system, we the Yahoo Flickr Creative Commons 100 Million Dataset (YFCC100m) (Thomee al. 2016). The dataset is drawn from about 200,000 videos (2,500 total hours) of various lengths and covering a diverse range of semantic sound categories. By splitting on video uploader, select 1,600 videos for training, and use the remaining videos for validation and test. We extract 5-second clips with a hop size of second, resulting in around 7.2 million clips. Clips consist of 5-second audio waveform sampled at 16 kHz and video frames (f), where each frame 128 128 x 3 RGB image.

---

Published as a conference paper at ICLR 2021
Our goal is to train our system completely unsupervised, but sought to reduce the proportion of videos with no on-screen sounds. We thus created a filtered subset D, of YFCC100m of clips with a high audio-visuai coincidence probability predicted by an unsupervised audio-visuaa coincidence prediction model (Jansen et al. 2020) trained on sounds from AudioSet (Gemmeke et al. 2017). The resulting selection is noisy, because the coincidence model is not perfect, and clips that have high audio-visual coincidence may contain both on-screen and off-screen sounds, or even no on-screen sounds. However, this selection does increase the occurrence of on-screen sounds, as shown below. The final filtered dataset consists of all clips (about 336,000) extracted from the 36,000 highest viitull coincidence scoring videos. The threshold for filtering was empirically set to keep fair amount of diverse videos while ensuring that not too many off- off-screen-only clips were accepted.
To evaluate the performance of the unsupervised filtering and our proposed models, and to experiment with small amount of supervised training data, we obtained human annotations for 10,000 unfiltered training clips, 10,000 filtered training clips, and 10,000 filtered validation/test clips. In the annotation process, the raters indicated "present" or "not present" for on-screen and off-screen sounds. Each clip is labeled by 3 individual raters, and is only considered on-screen-only or off-screen-only if raters are unanimous. We constructed clips, and an off-screen--nly subset with 3,681 training, 836 validation, and 370 test clips.
screen--e-nlly subset with 836 training, 735 validation, and 295 test
Based human annotations, estimate that for unfiltered data 71.3% of clips contain both on-and- off-screen sounds, 2.8% contain on-screen-only sounds, and 25.9% only off-screen sounds. For the filtered data, 83.5% of clips contain on-screen and off-screen sounds, 5.6% of clips are on-screen- only, and 10.9% are off-screen-only. Thus, the unsupervised filtering reduced the proportion of off-screen-only clips and increased the proportion of clips with on-screen sounds.
4.2 TRAINING
Both audio and visual embedding networks were pre-trained on AudioSet (Gemmeke et al.) 2017) for unsupervised coincidence prediction (Jansen et al., (020) and fine-tuned on our data (see Appendix
A.3.1 for ablation), whereas the separation network is trained from scratch using MixIT (2) on
mixtures of mixtures (MoMs) from the audio of our data. All models are trained on 4 Google Cloud
TPUs (16 chips) with Adam (Kingma & Ba 2015), batch size 256, and learning rate 10-1.
To train the overall network, we construct minibatches of video clips, where the clip's audio is either a single video's soundtrack ("single mixture" example), or a mixture of two videos' soundtracks ("MoM" example): NOn (noisy-labeled on-screen). SOff (synthetic off-screen-only), LOn (human- labeled on-screen-only), and LOff (human-labeled off-screen-only). For all MoM examples, the second audio mixture is drawn from different random video in the filtered data. Unsupervised minibatches consist of either 0% or 25% SOff examples, with the remainder as NOn. NOn examples always MoMs, and SOff examples MoM uses video clip frames and audio from the filtered high-coincidence subset of our data, Dy, and SOff MoMs combine video frames of filtered clip with random audio drawn from the dataset Dy,
evenly split between single mixtures and MoMs. A NOn
Semi-supervised minibatches additionally include LOn and LOff examples. Half of these examples in the minibatch are single-mixture examples, and the other half are MoM examples. LOn and LOff examples are constructed in the manner as NOn, except that the corresponding video clip is drawn from unanimously human-labeled on on-screen-onf videos and unanimously numan-labeled off- screen-only videos, respectively. We experiment with using 0% or 25% SOff examples: (NOn, SOff) proportions of (50%, 0%) or (25%, 25%), respectively, with the remainder of the minibatch evenly split between 1.nn single-mixture, L.On MoM, LOff single-mixture, and LOff MoM.
Classification labels ym for all separated sources &m in SOff and LOff examples are set to O. For NOn and LOn examples, we set the label for each separated source as the first row of the MixIT mixing matrix (2): 3m Al,m The MixIT separation loss (2) is used for all MoM example types.
4.3 EVALUATION
All evaluations use labeled test videos, which have been unanimously labeled as containing either only on-screen only off-screen sounds. Using this data, we construct four evaluation sets: on-screen single mixtures, off-screen single mixtures, on-screen MoMs, and off-sereen MoMs. The single-mixture evaluations consist of only data drawn from the particular label, either on-screen or
7

---

Published as a conference paper at ICLR 2021
ofr-sereen. Each on-screen (off-screen) MoM consists of an on-screen-only coff-screen-only video clip, mixed with the audio from another random clip, drawn from the seffsereen-nlly examples.
4.3.1 ON-SCREEN DETECTION
Detection performance for the on-screen classifier is measured using the area under the curve of the weighted receiver operator characteristic (AUC-ROC). Specifically, set the weight for each source's prediction equal to the linear ratio of source power to input power, which helps avoid ambiguous classification decisions for inactive or very quiet sources. For single-mixture evaluations, positive labels are assigned for all separated sources from on-screen-only mixtures, and negative labels for all separated sources from off-screen-only mixtures. For on-screen MoM evaluations, labels for separated sources from on-screen MoMs are assigned using the first row of the oracle MixIT mixing matrix, and negative labels are assigned sources separated from off-screen MoMss
4.3.2 SEPARATION
Since we do not have access to individual ground-trutt reference sources for our in-the-wild data, we cannot evaluate the per-source separation performance. The only references we have are mixtures. Thus, we compute an estimate of the on-screen audio by combining the separated sources using classifier predictions: oon = >M &m. For on-screen single mixture and MoM evaluations, measure scale-invariant SNR (SI-SNR) (Le Roux et al., 2019), between z00 and the reference on-screen-only mixture x (on)), SI-SNR measures the fidelity between a target t E R²" and an estimate f E IK" within an arbitrary scale factor in units of decibels:
||at|2 ||at
ITi
SI-SNR(t, 8) = 10 10g10
a = argmin, ||at - f||
f|I2"
||t||2"
To measure the degree to which AudioScope rejects off-screen audio, we define the off-screen suppression ratio (OSR), which is the ratio in decibels of the power of the input mixture to the power of the on-screen estimate zon, We only compute OSR for off-screen evaluation examples where the input mixture only contains off-screen audio. Thus, higher OSR implies greater suppression of off-screen sounds. The minimum value of OSR is 0 dB, which means that snnl is equal to the input mixture, which corresponds all on-sereen classifier probabilities being equal to 1.
In some cases, SI-SNR and OSR might yield infinite values. For example, the estimate 9 may be zero, in which case SI-SNR 7 is oo dB. This can occur when the input SNR of an on-screen mixture in a MoM is very low and none of the separated sources are assigned to it by MixIT. Conversely, if the estimate perfectly matches the target, SI-SNR can yield a value of oe dB, which occurs for on-screen single mixture evaluation cases when the separated sources trivially add up to the on-screen input due to mixture consistency the separation model. For off-screen examples, OSR can also be infinite the separation model achieves perfect off-screen suppression by predicting zero for zon To avoid including these infinite values, we elect to measure median SI-SNR and OSR.
5 RESULTS
Results are shown in Table I Note that there is a trade-off between preservation of on-sereen sounds, as measured by SI-SNR, and suppression of off-screen sounds, as measured by OSR: higher on-screen SI-SNR on on-screen examples generally means lower OSR on off-screen examples. Different classification losses have different operating points: for MoMs, compared using the exact cross-entropy loss, models trained with active combinations or multiple instance loss achieve lower on-screen SI-SNR, while achieving more suppression (higher OSR) of off-screen sounds. Exact cross-entropy models achieve higher AUC-ROC for single mixtures and MoMs, and achieve better reconstruction of on-screen single mixtures at the expense of less rejection of off-screen mixtures.
Training only with the noisy labels provided by the unsupervised coincidence model (Jansen et al. 2020) achieves lower AUC-ROC compared to the semi-super condition that adds a small amount of
examples. Semi- superviuee and unsupervised models achieve comparable screen SI-SNR, but semi-supervised models achieve better oef-screen suppression. For example, the best on-screen SI-SNR for unsupervised and sepe-supe is 8.0 dB and 7.3 dB, respectively,
8

---

Published as a conference paper at ICLR 2021
Table 1: Evaluation results for umanimously-annotaa on-screen and off-screen mixtures. Training uses unsupervised or semi-supervised examples, with either 0% or 25% synthetic off-screen (SOff) examples (Section 4.2). Cross-entropy (CE) losses include active combinations (AC), multiple instance (MI), and exact. Note that "MixIT*" indicates SI-SNR of oracle estimate derived using MixIT with reference mixtures, and z the on-screen estimate produced by mixing separated sources with classifier probabilities. On-screen MoMs have median input SI-SNR of 4.4 dB.
Single mixture
Mixture of mixtures
On: SI-SNR (dB)
Off: (dB)
SI-SNR (dB)
OSR (dB)
Training
SOff
loss
M
AUC
M
6.00
Unsup 0%
AC
0.58
13.5
0.77
10.5
6.3
9.4
Unsup
0%
MI
0.55
11.9
3.6
0.75
10.1
5.3
10.8
Unsup
0%
Exact
0.62
00
36.6
0.5
0.81
10.6
8.0
5.3
Unsup
12.1
9.3
Unsup
25%
MI
0.62
13.6
4,0
0.78
9.6
6.1
9.6
Unsup
25%
Exact
0.64
41.0
0.6
0.81
10.6
7,5
4.5
Semi 0%
AC
0.71
00
14.8
6.6
0.82
10.4
6,1
14.1
Semi 0%
Exact
0.73
32.8
4.5
0.81
10.1
7.3
10.7
Semi 25%
AC
0.79
6.7
54.3
0.78
10.0
3.4
61.8
2.0
60.1
Semi
25%
Exact
0.83
53.9
0.81
10.0
2.4
61.5
while OSR is 5.3 dB and 10.7 dB. Using 25% synthetic ocf-scree particularly shifts the behavior of semi-supervised models by biasing them towards predicting lower probabilities of on-screen. This bias results in lower on-screen SI-SNR, yet very strong off-screen rejection (i.e. very large OSRs).
Ireupercied woett MieT
Linuperriied PuNet,
Unsupersines eeart
Semi--uperiieed roort, NadT
sepp--upeerudddddd evort,
Samii superdoed mcart
25 Fpuc S4.
25
33
25
25
40
33
Figure 3: Scatter plots of input SI-SNR versus on-screen SI-SNR for on-screen evaluation examples, and mixture power in dB versus OSR. The settings for models are SOff 0% and exact CE loss, and colormap indicates density of points.
Figure shows scatter plots of input SI-SNR versus SI-SNR of MixIT or 2 on-screen estimates. From these plots, is clear that the models tend improve on-screen SI-SNR more often than not, and that these improvements are most significant around 10 dB input SI-SNR. Note that for MixIT, a number points have SI-SNR oc, which happens when MixIT assigns all separated sources when AudioScope achieves excellent off-screen
to the off-screen mixture. OSR is sometimes suppression by predicting nearly 0 for the on-screen audio from off off-screen-only input. To provide a sense of the qualitative performance of AudioScope, include visualizations of best, worst, and typical predictions in the appendix, and the supplementary material contains audio-visual demos
To benchmark AudioScope against other audio-visun separation approaches and measure perfor- mance on mismatched data, we evaluate on existing audio-visual separation test sets in Appendix A.2
We also performed number of ablations for AudioScope, described in Appendix A.3
CONCLUSION
In this paper we have proposed the first solution for training an unsupervised, open-domain, audio- visual on-screen separation system, without reliance prior class labels or classifiers. We demon- strated the effectiveness of our system using a small amount of human-labeled, in-the-wild videos. recipe for these will be available on the project webpage: https: / htt:///udioon audioscope gittub io In future work, we will explore more fine-grained visual features, especially synchrony, which we expect will be especially helpful when multiple instances of the same object present in the video. We also plan to use our trained classifier to refilter YFCC100m to get better noisy labels for the presence of on-screen sounds, which should further improve the performance of the system.

---

Published as a conference paper at ICLR 2021
REFERENCES
Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. In Proc. European Conference on Computer Vision (ECCV), 2020.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn In Proc. IEEE International Conference on Computer Vision (ICCV), pp. 609-617, 2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. International Conference on Learning Representations (ICLR), 2015.
Lukas Drude, Daniel Hasenklever, and Reinhold Hacb-Umbach. Unsupervised training of deep clustering model for multichannel blind source separation. In Proc. IEEE International Conference Acoustics, Speech, and Signal Processing (ICASSP), pp. 095-699, 2019.
Ariel Ephrat, Inbar Mosseri, Oran Lang. Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: a speaker-independen audio-visual model for speech separation. ACM Transactions Graphics (TOG), 37(4):1-11, 2018.
Chuang Gan, Deng Huang, Hang Zhao, Joshua Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In Proc. IEEE International Conference Computer Vision (CVPR), pp. 10478-10487, 2020.
Ruohan Gao and Kristen Grauman. Co-separating sounds visual objects. Proc. IEEE Interna- tional Conference on Computer Vision (CVPR), pp. 3879-3888, 2019.
Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proc. European Conference on Computer Vision (ECCV), pp. 35-53, 2018.
Jort F Gemmeke, Daniel W. Ellis. Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 776-780, 2017.
Rohit Girdhar and Deva Ramanan. Attentional pooling for action recognition. In Advances in Neural Information Processing Systems, pp. 34-45, 2017.
David Harwath, Adria Recasens, Dídae Surís, Galen Chuang, Antonio Torralba, and James Glass. Jointly discovering visual objects and spoken words from raw sensory input. In Proc. European Conference on Computer Vision (ECCV). pp. 649-665,20 2018
John Hershey and Michael Casey. Audio-visual sound separation via hidden Markov models. In Advances in Neural Information Processing Systems, pp. 1173-1180, 2002.
John R Hershey and Javier R Movellan. Audio vision: Using audio-visual synchrony to locate sounds. In Advances Neural Information Processing Systems, pp. 813-819, 2000.
Jen- Jen-Cheng Hou, Syu-Siang Wang. Ying-Hui Lai, Yu Tsao, Hsiu-Wai Chang, and Hsin-Min Wang- Audio-visual speech enhancement using multimodal deep convolutional neural networks. JEEE Transactions on Emerging Topics in Computational Intelligence, 2(2):117-128 2018.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1=Xxv:170400486) 2017.
Di Hu, Zheng Wang, Haoyi Xiong, Dong Wang, Feiping Nie, and Dejing Dou. Curriculum audiovisual learning. arXiv preprint arXiv:2001. 00t:::00000091444 2020.
Aren Jansen, Daniel PW Ellis. Shawn Hershey, R Channing Moore, Manoj Plakal, Ashok C Popat, and Rif A Saurous. Coincidence, categorization, and consolidation: Learning recognize sounds with minimal supervision. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). pp. 121-125, 2020.
10

---

Published as a conference paper at ICLR 2021
Iya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, and John R. Hershey. Universal sound separation. In Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Interna- tional Conference on Learning Representations (ICLR), 2015.
Qiugiang Kong, Yuxuan Wang, Xuchen Song, Yin Cao, Wenwu Wang. and Mark D Plumbley. Source separation with weakly labelled data: An approach to computational auditory scene analysis. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 101-105, 2020.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Advances in Neural Information Processing Systems, pp. 7763-7774, 2018.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey. SDR-hall baked well done? In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 626-630, 2019.
Oded Maron and Tomás Jozano-Pérezz A framework for multiple-instene instance learning. In Advances in Neural Information Processing Systems, pp. 570-576, 1998.
Daniel Michelsanti, Zheng-Hut Tan, Shi-Xiong Zhang, Yong Xu, Meng Yu, Dong Yu, and Jesper Jensen. An overview of feep-learning-bede -bssed audio-visual speech enhancement and separation. arXiv preprint Ttt.:2000.095500 2020.
Tsubasa Ochiai, Marc Delcroix, Yuma Koizumi, Hiroaki Ito, Keisuke Kinoshita, and Shoko Araki. Listen to what you want: Neural network-bae based universal sound selector. In Proc. Interspeech, 2020.
Andrew Owens and Alexci A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proc. European Conference Computer Vision (ECCV), pp. 631 1488 2018.
Pishdadian, G. Wichern, and J. Le Roux. Finding strength weakness: Learning to separate sounds with weak supervision. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 2386-2399 2020.
Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Torralba. Self- supervised audio-visual co-segmentation. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 2357-2361. IEEE, 2019.
Prem Seetharaman, Gordon Wichern, Jonathan Le Roux, and Bryan Pardo. Bootstrapping single- channel source separation via unsupervised spatial clustering stereo mixtures. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 356-360, 2019.
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proc. IEEE International Conference Computer Vision (CVPR), pp. 4358-4366, 2018.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data multimedia research. Communications of the ACM, 59(2):44-33 2016.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. audio-visual event localization in unconstrained videos. In Proc. European Conference Computer Vision (ECCV), pp. 247-263, 2018.
Efthymios Tzinis, Shrikant Venkataramani, and Paris Smaragdis. Unsupervised deep clustering for source separation: Direct learning from mixtures using spatial information. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 81-85, 2019.
11

---

Published as a conference paper at ICLR 2021
Efthymios Tzinis, Scott Wisdom, John R. Hershey, Aren Jansen, and Daniel P. W. Ellis. Improving universal sound separation using sound classification. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 96-100, 2020.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 14(4):
1462-1469, 2006.
Scott Wisdom, John Hershey, Kevin Wilson, Jeremy Thorpe, Michael Chinen, Brian Patton, and Rif A. Saurous. Differentiable consistency constraints for improved deep speech enhancement, In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp.
900-904, 2019.
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron J. Weiss, Kevin Wilson, and John R. Hershey. Unsupervised sound separation using mixture invariant training. In Advances in Neural Information Processing Systems, 2020.
Scott Wisdom, Hakan Erdogan, Daniel Ellis, Romain Serizel, Nicolas Turpault, Eduardo Fonseca, Justin Salamon, Prem Seetharaman, and John Hershey. What's s all the fuss about free universal sound separation data? Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2021.
Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In Proc. IEEE International Conference on Computer Vision (CVPR), pp. 6292-6300, 2019.
Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus-plus net. In Proc. IEEE International Conference on Computer Vision (CVPR), pp.
2019.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proc. European Conference on Computer Vision (ECCV), pp. 570-586, 2018
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proc. IEEE International Conference on Computer Vision (CVPR), pp. 1735-1744, 2019.
Lingyu Zhu and Esa Rahtu. Separating sounds from a single image. arXiv preprint arXiv:2007.07984, 2020a.
Lingyu Zhu and Esa Rahtu. Visually guided sound source separation using cascaded opponent filter network. In Proc. Asian Conference Computer Vision, 2020b.
A APPENDIX
A.1 QUALITATIVE EXAMPLES
For range of input SNRs, Figure 4 shows best-case examples of separating on-screen sounds with AudioScope, while Figure shows failure cases. Figures 6 and show random examples at various SNRs, comparing the outputs of semi-supervised SOff 0% models trained with either exact cross entropy 4 or active combinations cross entropy (6). Figure o shows the outputs of the two models on random examples, and Figure 7 shows the outputs of the two models on 5 examples that have maximum absolute difference in terms of SI-SNR of the on-screen estimate.
The supplementary material includes audio-visual demos of AudioScope on single mixtures and MoMs with visualizations of MixIT assignments and predicted on-screen probabilities. For more examples please see:
12

---

Published as a conference paper at ICLR 2021
Video frame
On-screen aucio
Off-screen audio
Input audio mixture
Aitention map
On-screen estimate
Input SNA 37, 9.98
MASSS 37 Sae
Input SNA 29. 99.088
Input SNA 18 3dB
SI SNR 26 38B
Input 5WR 8,688
BLSVR 22.28B
Input SNR 7788
SISNR 20. 6d8
222
2222
Input SNR -15 248
SUSRR 14 30B
Input -23 4:8
SI SNR 1.148
Figure 4: Best cases for AudioScope separation of on-screen sounds under various SNR conditions.
13

---

Published as a conference paper at ICLR 2021
On-screen audio
Input mixture
Alcention map
On-screen
Input SAR -29 A48
D2212 ZE 2ISS 15
Input SNR -20 20.2888
Input SNR - 3-18
St S.S 17.5a8
Input SAR 7.B
Input SNR 5.588
SISNR 5. 5,048
Input SNA 20, SeB
CASSO 6...1. 10B
Input SARR 30. 8dB
SNR 66B
Figure Failure cases for AudioScope separation on-sereen sounds under various SNR conditions.
14

---

Published as a conference paper at ICLR 2021
Inpur SNA .14 9d8
SLSNI 6dt
Input SNA a 0.08
SISA 5,000
SISAR 1,360
Input SNIR -5 5,0488
SUSNAR 7000
ULSAR
Inpuut SNA -. 0.08
SISMR 0.200
SLSAR
Kopuir SMA 5.088
S1 SES 7.8000
Inpur Shi L.ods
SYSNNR
Impurt SMA 34, Ari8
SYSAR 2d0
Exact loss
Exact loss
AC loss
AC logs
Figure Comparison of random examples separating on-screen sounds under various SNR
conditions using either exact cross entropy or active combinations cross entropy (6) as the classification loss.
15

---

Published as a conference paper at ICLR 2021
Video frame
On-screen autio
off-screen audio Input audio mixture Attention map
On-screen estimate Altention map
On estimate
Inpue SAR 8. 1.88
S.SM 3.3d8
Inpur SNR -1.78 7.7788
S1SAR 3.500
Ingutt SNR .0o8
SISAR 7.800
Inpuit 5MA 1.2288
SL-SAR 1d18
SHAAR 23. 9H1
Inpour SMR 707888
SISNI 7d0
Exact loss
Exact loss
AC less
AC lass
Figure 7: Comparison of examples with maximum absolute performance difference under various SNR conditions using either exact entropy active combinations entropy (6) the classification loss.
16

---

Published as a conference paper at ICLR 2021
A.2 EVALUATION ON MISMATCHED DATA
To evaluate the generalization capability of AudioScope and facilitate a comparison to prior works, evaluated our model using test data from an audio-visual speech enhancement task (Hou et al. 2018) as well as an undio-visual task derived from single-source subset of AudioSet (Gao et al. 2018). In both cases, the evaluation is on a restricted domain, and the prior methods used both matched and supervised training on that domain. In contrast, the AudioScope model is trained on open-domain YFCC100m videos using unsupervised training. For all evaluations we use the unsupervised AudioScope model using 0% SOff and active combinations loss.
A.2.1 EVALUATION ON AUDIO- VISUAL SPEECH ENHANCEMENT
Since our method can be used to separate on-screen sounds for arbitrary classes of sound, to compare to existing approaches we evaluate the trained AudioScope model on the more restricted domain of audio-visual speech enhancement. To that end, we used the Mandarin sentences dataset, introduced by Hou et al. (2018). The dataset contains video utterances of Mandarin sentences spoken by a native speaker. Each sentence is unique and contains 10 Chinese characters. The length of cach utterance is approximately 3 to 4 seconds. Synthetic noise added to each ground truth audio. Forty such videos used the official testing set. For evaluation regard the speech of the filmed speaker as the on-screen sounds and the interference as off-screen sounds. Thus, we can compute quality metrics for the on-screen estimate while comparing to speech enhancement methods. To compare to previously-publisher numbers, we use iignalto-distortion ratio (SDR) (Vincent et al. 2006), which measures tignil-to-noiee ratio within linear filtering of the reference signal.
Table 2 shows the comparison between Hou et al. (2018). Ephrat et al. (2018), AudioScope .:om (on-screen estimate using predicted on-screen probabilities) AudioScope source with max Im (use separated source with highest predicted on-screen probability), AudioScope best source (oracle selection of the separated source with the highest SDR with on on-screen reference), and AudioScope MixIT* (on-screen estimate using oracle binary on-screen weights using references). Note that the AudioScope models are trained with mismatched open-domain training data, whereas the others were trained on matched speech enhancement data. It can be seen that although non-oracle AudioScope estimates do not advance on state-of-the-ar performance of speech enhancement-speecif methods, the oracle AudioScope estimates improve over (Hou et al. 2018). Thus AudioScope show promising results on this challenging data which is not explicitly represented in its open-domain training set. training set, perhaps by fine-tuning, AudioScope could
We believe that by adding such data improve its performance significantly on this more specific task, which we leave for future work.
Table 2: Audio-visual enhancement results on the Mandarin test set (Hou et al. 2018).
Method
In- nn-Domain Supervised
SDR
STOI
Hou et al.
2018
2.8
0.66
Ephrat et al. 2018
6.1
0.71
AudioScope fon
X
X
2.5
0.59
AudioScope source with max Im AudioScope best source (oracle)
X
X
2.3
0.58
X
X
3.2
0.60
AudioScope MixIT* (oracle)
X
X
3.4
0.61
A.2.2 EVALUATION ON AUDIOSET-
We evaluated AudioScope on the musical instrument portion of the AudioSet-SingleSource dataset (Gao et al. 2018), which is small number of clips from AudioSet (Gemmeke et al.) 2017) that have been verified by humans to contain single sources. We use the same procedure as Gao & Grauman (2019) to construct a MoM test set, which creates 105 synthetic mixtures from all pairs of 15 musical instrument classes. For each pair, audio tracks are mixed together, and we perform separation twice for each pair, conditioning on the video for each source. The results are shown in table
The non-oracle AudioScope methods perform rather poorly, but the oracle methods, especially M (which matches the MixIT training loss), achieve state-of-the-art performance compared to
17

---

Published as a conference paper at ICLR 2021
Table Audio-visual separation results on AudioSet-SinlleSoure musical instruments test (Gao & Grauman 2019).
Method
In-Domain
Supervised
SDR
SIR
SAR
Sound-of- of-Pislls (Zhao et al. 2018)
1.7
3,6
11.5
AV-MIMI. (Gao al. 20188
1.8
Co-Separation (Gao Graumann 2019
4.3
7.1
13.0
AudioScope 2.nn
X
0.4
2.7
11.4
AudioScope source with max Dm AudioScope best source (oracle)
X
-0.9
2.8
7.9
X
X
4.6
9.9
12.1
AudioScope MixIT* (oracle)
X
5.7
8.4
12.5
methods form the literature. This suggests that the on-screen classifier is less accurate on this data. Also, mixing the predicted AudioScope sources using the probabilities the on-screen classifier may be suboptimal, and exploring alternative mixing methods to estimate on-screen audio an avenue for future work. Fine-tuning on data for this specific task could also improve performance, which we also leave for future work.
A.2.3 EVALUATION MUSIC
We also evaluated AudioScope MUSIC (Zaao et al. 2018), which includes video clips of solo musical performances that have been verified by humans to contain single sources. We use the same procedure Gao & Grauman (2019) construct a MoM test set, which creates 550 synthetic mixtures from all 55 pairs of 11 musical instrument classes, with 10 random 10 second clips per pain For each pair, the two audio clips mixed together, and perform separation twice for each pair, conditioning on the video for each source. The results are shown in table
Table 4: Audio-visual separation results MUSIC test set (Zhao al.
2018
Method
In-Domain
Supervised
SDR
SIR
SAR
Sound-of-Pixee P-iieels (Zhao al. 2018
5.4
11.0
9.8
Sound-of-Motioon of-ooooon-s (Zhao et al. 2019
4.8
11.0
8.7
MP-Net Xu al. (2019)
5.7
11.4
10.4
Co-Separation (Gao Grauman, 2019
V
7.4
13.7
10.8
Cascaded Opponent Filter (Zhu Rahtu 2020b
10.1
16.7
13.0
A(Res-50, att) + S(DV3P) (Zuu & Rahtu 2020a
9.4
15.6
12.7
A(Res-50, class.) + S(DV3P) Zhu & Kahtu 2020a
10.6
17.2
12.8
AudioScope gen
-0.5
2.8
11.2
AudioScope source with max fm
X
-2.0
3.3
7.6
AudioScope best source (oracle) AudioScope MixIT* (oracle)
X
X
7.1
14.9
12.5
8.8
13.0
13.1
We see a similar pattern compared to the results for AndioSet-SingleSoure in Table non-oracle methods that use the predicted on-screen probability 2m do not perform very well. However, oracle selection of the best source, or oracle remixing of the sources, both achieve better performance than number of recent specialized supervised in-domain systems from the literature, though they do not achieve state-of-the. of-t----rr-tt performance. These results seem to suggest that the predictions im are less accurate for this restricted-domain task, but the excellent oracle results suggest potential. In particular, non-oracle performance could improve if the classifier were more accurate, perhaps by fine-tuning, Also, there may be better ways of combining separated sources together to reconstruct on-screen sounds.
Gao & Grauman 2019 did not provide the exact clip timestamps they used. We used a sliding window of 10 seconds with hop of seconds, and randomly selected 10 of these.
18

---

Published as a conference paper at ICLR 2021
A.3 ABLATIONS
We performed a number of ablations on AudioScope. The following subsections show the results of number of ablations using either unsupervised semi-supervised training. All models for these ablation use 0% SOff examples and the active combinations loss 6
A.3.1 AUDIO AND VIDEO EMBEDDINGS
Table 5 shows the results of various ablations involving audio and video embeddings in the model.
Table 5: Ablations related to audio and video embeddings.
Single mixture
Mixture of mixtures
On: SI-SNR
OIf: OSR
On: S1-SNR
Off: OSR
Training
Ablation
AUCC
ge0
200
AUC
MixIT
200
Unsup
0.58
13.5
2.5
0.77
10.5
6.3
9.4
Unsup
conditioning sepuration
Unsup
Finh. netwarks from scratch
0.45
8,9
7.0
0.61
10.6
2.4
13.4
Unsup
glubal video and audio emb. classifier
Unsup
No global video emi.. to classifier
0.57
13.3
2.3
0.77
10.6
6.3
9.8
Semi
0.71
14.8
6.6
10.4
Semi
No video conditioning for separation
0.65
12.8
12.3
0.77
10.4
5.3
19.7
Semi
Einh. networks from scratch
0.48
6.3
12.7
0.59
9.7
16.4
Semi
No global video and audio emb. to classifier
0.69
11.1
14.7
0.78
10,1
4.5
21.4
Semi
Na global video emb. to classifier
0.66
11.7
10.0
0.77
10.2
16.3
First, notice that removing video conditioning for the separation model reduces on-screen SI-SNR by 2 dB on single mixtures and 0.9 dB on MoMs, with negligible slight improvement OSR. Thus, we can conclude that visual conditioning does have some benefit for the model.
Next, we consider training the audio and video embedding networks from scratch, instead of using the coincidence model weights pretrained using AudioSet (Jansen et al. 2020). Training from scratch quite detrimental, as AUC-ROC decreases by minimum of 0.13 and maximum 0.23 across miggll and wttpeerriiidd/mm-suupeeeeeeeeeee performance suffers, with on-screen SI-SNR dropping by multiple for all conditions.
conditions, Furthermore, separation
Finally, we consider removing the global video embedding, both the global video embedding and audio embeddings, from the input of the on-screen classifier. This results in equivalent or slightly worse AUC-ROC, with equivalent worse on-screen SI-SNR. For unsupervised training, removing both embeddings at the classifier input improves on-screen SI-SNR a bit (0.5 dB for single mixtures, 0.6 dB for MoMs) with slight drop in OSR, though for semi-supervised on-screen SI-SNR drops by 3.7 dB for single mixtures and 0.5 dB for MoMs. Overall, the best result achieved by including these embeddings the classifier input.
A.3.2 ATTENTIONAL POOLING
We tried decreasing the embedding dimension from 256 128, as well replacing the attentional pooling with mean pooling for audio sources, video frames, or both. The results are shown in Table
Table 6: Ablations related to attentional pooling run for the unsupervised setting.
Single mixture
Mixture of mixtures
On: SI-SNR
OSR
On: SI-SNR
Off: OSR
Training
Ablation
AUC
gen
gen
AUC
MixIT
gmm
geu
Unsup
0.58
13.5
2.5
0.77
10.5
6.3
9.4
Unsup
embedding dimension 128
0.56
Unsup
No attentional pooling for audio sources
0.59
14.3
1.9
0.78
10.3
6.4
7.5
Unsup
attentional pooling video
0.57
Unsup
No attentional pooling for audio and video
0.56
12.8
1.8
0.76
10.5
6.3
7.7
Decreasing the embedding dimension reduces performance, dropping on-screen SI-SNR by 1.4 dB on single mixtures and 0.6 dB on MoMs, also with reduction in OSR. Replacing attentional pooling with mean pooling generally does not change AUC-ROC on-screen SI-SNR that much, but does result
19

---

Published as a conference paper at ICLR 2021
in OSR reduction of least 0.6 dB for single mixtures and 1.7 dB for MoMs. Thus, attentional pooling seems to have a beneficial effect in that improves off-screen suppression, with equivalent classification and on-screen separation performance.
A.3.3 DATA FILTERING
As described in section 4.1 we use unsupervised audio-visual coincidence model (Jansen al. 2020) to filter training videos for on- on-screen sounds. To ablate the benefit of this filtering, we tried using different combinations of filtered and unfiltered data for NOn examples, as described in section
which uses filterd data for both on-screen and off-screen mixtures. Filtered data has the advantage of less noisy on-screen labels, but the disadvantage that it lacks the variety of unfiltered data, being
only 4.7% of the unfiltered data.
Table 7: Ablations for different data configurations.
Single
Mixture mixtures
On: SI-SNR
Off: OSR
On: SI-SNR
Off: OSR
Training
Ablation
AUC
2
AUC
MixIT
g
Unsup
0.58
13.5
2.5
0.77
10.5
6.3
Unsup
Filtered on-screen, unfiltered off-screen
0.60
11.5
3.4
0.78
10.7
5.7
11.0
Unsup
Unfiltered
unfiltered off screen
0.60
12.6
2.5
0.77
10.5
6.7
Unsup
Unfiltered on-sereen, filtered off-screen
0.63
15.3
2.3
0.79
10,7
7.0
5.6
Semi
0.71
14.8
Semi
Filtered on on-screen, unfiltered off screen
0.68
11.1
7.1
0.78
10.2
5.3
13.1
Semi
Unfiltered on-sereen, unfiltered sor-ss
0.66
13.5
10,4
Semi
Unfiltered on sereeeeen filtered off screen
0.65
14.8
7.8
0.74
10.4
5.6
8.9
The results are shown in Table 7
For unsupervised training, unfiltered on-sereen with filtered off-screen achieves improved performance in terms of AUC-ROC and on-screen SI-SNR, yet OSR decreases for MoMs. This suggests that in the absence of cleanly-labeled on-screen videos, larger amount of data with noisier labels is better compares to smaller amount of data with less noisy labels. However, for semi-supervised training that includes small amount cleanly-labeled on-screen examples, AUC-ROC is consistently worse for all ablations, and on-screen SI-SNR and OSR are generally equivalent or worse for all ablations. Thus, these ablations validate that using filtered data for both on-screen and off-screen components of NOn examples with semi-supervised training achieves the best results overall.
A.3.4 NUMBER OF OUTPUT SOURCES
For all experiments in this paper, we generally used M = 4 output sources for the separation model, which is the maximum number of sources that it can predict. Here we see if increasing the number of output sources can improve performance. More output source slots provides separation model with more flexibility in decomposing the input waveform, yet the drawback is that the model may over-separate (i.e. split sources into multiple components), and there is more pressure on the classifier to correctly group components of the on-screen sound together. The results are shown in Table
Table 8: Ablations for number of max output sources.
Single mixture
Mixture of mixtures
Off
Training
Ablation
AUC
goo
go0
AUC
MixIT
jam
j.m
Unsup
0.58
13.5
2,5
0.77
10.5
6.3
9.4
Unsup
output sources
9.6
2.1
10.3
Unsup
output sources
0.52
6.9
4.5
0.69
11.1
3.6
10.1
Semi
0.71
14.8
6.6
0.82
10.4
14.1
Semi
0.67
7.9
12.4
0.76
10.8
4.0
20.4
Semi
8 output sources
0.67
6.5
18.8
0.77
10.9
2.6
24.6
For unsupervised training, increasing the number of output sources generally degrades AUC-ROC and on-screen SI-SNR, while boosting OSR a bit. Note that the MixIT improves for MoMs with 8 output sources (10.5 dB 11.1 dB), which suggests the greater flexibility of the model, yet the
20

---

Published as a conference paper at ICLR 2021
on-screen estimate zo0 is quite a bit worse (3.6 dB), also compared on-screen SI-SNR for 4 output sources (6.3 dB).
For semi-superriied training. MixIT performance also improves with more output sources, but AUC-ROC and on-screen SI-SNR decrease, suggesting the increased pressure on the classifier to make correct predictions for more, and potentially partial, sources. OSR increases with more output sources, which suggests the classifier biases towards predicting Os more often. Thus, increasing the number of sources shifts the operating point of the model away from separating on-screen sounds and towards suppressing off-screen sounds.
A.3.5 BASELINE SEPARATION MODEL
We also trained two-output baseline separation models without the on-screen classifier, where the first estimated source is the on-screen estimate j.s with training target of on-screen audio, and the second estimated source the off-screen estimate goff with training target of off-screen audio. These models were trained with without video conditioning, using the negative SNR loss B). The training data is exactly the same as in Table with 0% SOff.
Table 9: Results for baseline two-output separation model without on-screen classifier.
Single mixture
On-screen
Off-screen
Training
Video cond.
SI-SNR son
ISR .on
ISR 5off
ISR f
ISR 2off
Predict input for gon and O for goff
00
0.0
00
0.0
00
Predict 1/2 input for 4on and goff Predict 0 for zon and input for 2off
OC
3.0
3.0
3.0
NaN
[c
0.0
0c
0.0
Unsup
66.2
6.0
6.0
6.0
6.0
Unsup
29.7
0.2
27.3
2.4
6.7
Semi
-18.0
51.1
0.0
51.2
0.0
Semi
18.8
10.5
3.0
48.5
0.0
Mixture of mixtures
On-screen
Off-screen
Training
Video cond.
SI-SNR f
ISR soo
ISR .off
ISR 4on
ISR goff
Predict input for zon and for goff
4.4
0.0
00
0.0
0O
Predict 1/2 input for foo and gotf
4.4
3.0
3.0
3.0
3.0
Predict 0 for zon and input for .off
NaN
0c
0.0
0c
0.0
Unsup
4.4
6.0
6.0
6.0
6.0
Unsup
4.1
5.8
3.7
9.5
1.7
Semi
-19.7
53.3
0.0
54.2
0.0
Semi
-5.3
29.5
0.3
53.3
0.0
Table 9 shows the results in terms of the same metrics used in Tables T except that instead of "off-screen rejection ratio (OSR)", we report "input-to-souu soputt-o-souuce ratio (ISR)" (1.e. 10 of the ratio of input power to estimated source power) for each of the two output sources. High ISR means that the source power is lower compared
input power. Note that ISR sou off-screen single-mixtures and MoMs is equivalent to OSR. Table also includes several trivial baselines with expected scores.
First, notice that none of these models approach the performance of separation models that include the on-screen classifier, shown in Table Second, the unsupervised and semi-supervised models here achieve distinctly different operating points. Without vidco conditioning, the unsupervised model achieves trivial solution, nearly equivalent just outputting 1/2 the input mixture for each estimated source. Adding video conditioning for the unsupervised model actually reduces single-mixture performance bit (66.2 dB to 29.7 dB).
The semi- supervisd model without video conditioning is very poor at single-mixture on-sereen SI-SNR (-18.0 dB), yet achieves quite high single-mixture OSR (51.1 dB). As indicated by the ISRs,
21

---

Published as a conference paper at ICLR 2021
the model tends to prefer nearly-zero on-screen estimates, which may be due to the additional cleanly- labeled off-screen examples provided during training. For the sileo-conditioned semi-supervised model, single-mixture on-screen SI-SNR improves by quite a lot (-18.0 dB to 18.8 dB), but on-screen SI-SNR performance for on-screen MoMs is abysmal (-19.7 dB without visual conditioning, -5.3 dB with visual conditioning).
Overall, we can conclude from these baselines that simply training a two-output separation model with on-screen and off-screen targets, even with visual conditioning, is not feasible approach for our open-domain and noisily-labelee data.
A.4 NEURAL NETWORK AOCORKKARRCTTEEOO ARCHITECTURES
We briefly present the architectures used in this work for the separation network M the audio embedding network M and the image embedding network M and referred in Sections 3.] 3.2
and respectively.
We present the architecture of the TDCN+ separation network in Table 10 The input to the separation network a mixture waveform with time samples and the output is a tensor containing the M estimated source waveforms & IR xT The input for the ith depth-wise (DW) separable convolutional block the summation of all skip-residual connections and the output of the previous block. Specifically, there the following skip connections defined w.r.t. their index i 0,..., 31:
0 - 8, - 16, 0 - 24, 8 - 16, 8 24 and 16 24.
Table 10: TDCN+ separation network architecture for an input mixture waveform corresponding to time-length of 5 seconds, sampled at 16 kHz. The output of the separation network M separated sources. The dilation factor for each block is defined as D; mm)d(i i 0, 31.
Multiples
Layer operation
Filter size
Stride
Dilation
Input shape
ConviD
40 x 256
20
1 80, 000
x1
Dense
256 x 256
256 x 4, 000
Dense
256 512
256 000
PReLU
1 x x 1
512 x 4, 000
Instance Norm
512 Separable
512 000
x32
DW Conv ID
3 x 512 Separable
D,
512 x 4, 000
PReLU
1
1
512 000
Instance Norm
x 512 Separable 512 x 256
512 x 4, 000
Dense
512 x 4, 000
Dense
1 256 M 256
256 4, 000
x1
Reshape
M 256 x 4, 000
Conv TonyTransposel1 ID
40 256
20
M 256 X 4, 000
In similar way, in Table 11 MobileNet v1 architecture (Howard et al., 2017) with different input tensors.
we define the image and audio embedding networks, which use the same
The extraction of each image embedding j 1.... ... 5 relies on the application of the image embedding network. M on top of each input video frame individually. Moreover, in order to extract the local video spatio-temporal embedding, (denoted with a in Table TT) for each input video frame and feed it through a dense layer order to By concatenating all these intermediate convolutional maps
extract the output of the 8 8 convolutional map
reduce its channel dimensions form the local spatio-temporal video embedding z as specificed in Section 3.3
On the other hand, we extract time- varying embedding Z for the mith separated source waveform by applying the audio embedding network M on overlapping audio segments and concatenating those outputs. The audio segments are extracted with an overlap of 86 windows or cquivalently 0.86 seconds. Specifically, for each segment, we extract the mel-spectrogram representation from 96 windows with a length of 25ms and hop size of 10ms forming the input for the audio embedding network as matrix with size 96 x 64, where 64 is the number of mel-features. After feeding this mel-spectrogram as an input to our audio embedding network M we extract the corresponding static length representation for this segment 2j. where j denotes the segment index.
22

---

Published as a conference paper at ICLR 2021
Table 11: Audio and image embedding network architectures for an input segment log-mel spec- trogram corresponding
0.96 seconds, sampled at 16 kHz and an input image represented as an RGB tensor with shape 128 x 128 x 3, respectively. The log-mel spectrogram input to the audio embedding network has number of input channels C and the input video frame to the image embedding network has number of input channels C.n. = 3. Each depth-wise (DW) convolution or regular convolution is followed by a batch normalization layer and a ReLU activation. denotes the layer from which the local 8 x spatial feature map is extracted, as described in Section 3.3
Input shape
Layer operation
Filter size
Stride
Audio network
Image network
Conv
3 x 3 x Cin x 32
2
96 x 64 x
128 x 128 x
DW Conv
x 3 32 Separable
48 32
32
64 64 32
Conv
1x 1 x 32 x 64
1
48 x 32 x 32
64 x 64 x 32
DW Conv
3 x 64 Separable
2
x 64
64 x 64 64
Conv
1 X 1 x 64 x 128
1
24 x 16 x 64
32 x 32 64
DW Conv
x 3 X 128 Separable
24 x 16 128
32 32 128
Conv
1 1 X 128 128
24 x 16 128
32 32 128
DW Conv
x 3 x 128 Separable
2
24 x 16 128
32 x 32 x 128
Conv
1 1 128 256
12 x 8 128
16 16 128
DW Conv
3 3x x 256 Separable
12 x 8 256
16 16 X 256
Conv
1x 256 256
12 8 256
16 16 256
DW Conv
3 x 256 Separable
2
12 x 8 x 256
16 16 x 256
Conv
1x 1 256 512
6 4 256
8 8 x 256
DW Conv
3 3 x 512 Separable
6 X 4 X 512
8 8 x 512
Conv
1 512 512
X 4 X 512
X 512
DW Conv
3 3 x 512 Separable
6 6 x 512
8 x 8 x 512
Conv
1x 1 512 512
6 512
x 512
DW Conv
3 x 512 Separable
6x 512
8 8 x 512
Conv
512 512
6x 512
X
DW Conv
3 x 512 Separable
6 512
8 8 X 512
Conv
1x 1 x 512 512
6 512
8 8 X 512
DW Conv
3 512 Separable
6 X 512
8 8 X 512
Conv
1 1 x 512 512
6 512
8 X 8 X 512
DW Conv
3 x 512 Separable
2
6 512
8 x 512
Conv
1 1 x 512 1024
3 512
4 X 4 x 512
DW Conv
x 3 1024 Separable
3 1024
4
1024
Conv
1 X 1 X 1024 x 1024
3 x 2 x 1024
4 X X 1024
Average Pooling
x x 1024
4
1024
Dense
1 x 1 x 1024 x 128
1 1 x 1024
1 X 1 X 1024
23

---

Published as a conference paper at ICLR 2021
A.5 HUMAN EVALUATION
To determine the subjective quality of AudioScope predictions, we performed another round of human annotation on-screen test MoM videos. The rating task is the same the used to annotate data, as described in Section 4.1 where raters were asked to mark the presence of on-screen sounds and off-screen sounds. All models for these evaluations are the same the base model used in Appendix A.3;; 0% SOff examples with active combinations loss ( Each example was annotated by raters, and the ultimate binary rating for each example determined by majority.
Table 12: Results of human annotation task on on-sereen MoM test set.
Method
% on-screen only
% off-screen only
% on-and-off-screen
% unsure
Unprocessed
24.1
2.7
67.4
5.9
Unsup gou
38.0
2.5
53.8
5.7
Unsup MixIT*
37.1
1.1
53.7
8.0
Semisup gon
37.6
2.3
54.0
6.1
Semisuu MixIT*
36.9
1.2
52.8
9.1
The results for the on-screcn MoM test set are shown in Table [2 We evaluated both the estimate foo computed by a weighted sum of the separated sources Sm with the predicted probabilities ÿm. as well as the oracle remixture of separated sources to match the on-screen and off-screen reference audios (denoted by MixIT*). In this case, notice that all methods improve the percentage of videos rated as on-screen-only from 25.7% to about 37% or 38% for all methods.
Overall, these human evaluation results suggest lower performance than the objective metrics in Table One reason for this is that the binary rating task is ill-suited towards measuring variable levels of off-screen sounds. That is, a video will be rated as on-screen only there is absolutely off-screen sound. However, even if there is quiet off- serenn sound present, artifacts from the separation, a video will rated as having off-screen sound. Thus, the proportion human-rated on-screen-only videos be interpreted the number of cases where the model did perfect job removing off-screen sounds.
We plan to new human evaluation tasks with metter-matched questions. For example, could ask raters to use a categorical scale, e.g. mean opinion score from to 5. Another idea is to ask raters score the loudness of on-screen sounds with respect off-screen sounds the bottom of the scale means on-screen sound is much quieter than off-screen sound, middle of the scale means on-screen sound equal in loudness off-screen sound, and top of the scale means on-screen sound is much louder than off-screen sound.
sliding scale, where
24

---

Published as a conference paper at ICLR 2021
A.6 PERFORMANCE ANALYSIS OF BEST MODELS
In Figure 8 we show the distributions of overall SI-SNR and SI-SNR improvement, as well as OSR for the best unsupervised and semi-supervised models. We have neglected outliers (including infinite values) in both axes in order to focus on the most common samples.
40
50
30
30
of--sreen supression (4BS
40
20
20
30
SI-SARI
10
20
On-ateeen
10
-30
-40
-40
Input SI-MR (d8)
Input SHSMR (dB)
Input mixture power (dB)
Unsupervised
Semi superviedd
Unsvpervised
Semi-superiised
Unsupervised
supervised
(a)
(b)
(c)
Figure 8: Distribution plots for the performance obtained by the best model in terms on-screen SI-SNR (Figure Sa) and SI-SNRi (Figure 8b) reconstruction and off-screen power suppression (Figure 8c The settings for the models are SOff 0% and active combinations (AC) cross-entropy loss.
In Figure [ for on-screen MoMs show the distribution of each performance metric for these models versus different ranges of input SI-SNRs lying between [-30.30 both for absolute on- screen SI-SNR (Figure 9a) and on-screen SI-SNR improvement (Figure (b). For off-screen test MoM videos, we plot the distribution of OSR for different ranges of input mixture power lying between [-40, 0JdB (Figure
For on-screen SI-SNR and SI-SNRi, notice that the performance of the unsupervised and semi- supervised models is similar except for the [-30, 20] dB range of input SI-SNR. In Figure 9c note that both models achieve OSR of at least 0 dB for 75% of examples, and thus suppress off-screen sounds for at least 75% of the test data.
25

---

Published as a conference paper at ICLR 2021
20
On-screen 5I-SAR (cB)
-40
Unsupervised
Semi-supervised
[-30,
[-20,
[-10.
1D]
[10. 20)
30)
Input
(a) On-screen reconstruction performance in terms of SI-SNR for on-sereen MoMs, for each input SI-SNR bucket.
40
20
Onrscreen Si-SNRi (0B)
-20
-60
Unsupervised
Semi-supervised
[-30,
[-20,
10]
30)
Input (dB)
(b) On-screen reconstruction performance in terms of SI-SNR improvement (SI-SNRi) for on-screen MoMs, for each input SI-SNR bucket.
40
Unsupervised Semi-supervised
Off-screen supression (dB)
30
20
10
[-40, 30]
(-30, 200
[-20, -10]
[-10, 0]
mixture (dB)
(c) Off-en power suppression (OSR) distribution for off-screen MoMs, each input mixture power bucket dB.
Figure 9: Distribution plots for the performance obtained by the best model under different ranges of input SI-SNR and input mixture powers, for both unsupervised and semi-supervised settings. For each distribution plot, depict the 25th, 50th and 75th percentiles with dashed lines. The settings for the models are SOff 0% and active combinations (AC) cross cross-entropy loss.
26

---

Published as a conference paper at ICLR 2021
A.7 ATTRIBUTIONS
Images in figures are resized stills with or without overlaid attention maps from the following videos.
Figure
"Whitethroat" by Rac CC-BY 2.0
Figure 2
LLccaador and Yellow Jumpsuit" by tenaciousme CC-BY 2.0
Figure
"Video of SiX hands on the piano" by superhua CC-BY
"Small waterfall" by Jay lamboli CC-BY 2.0
"Roval lap" by meipseric CC-BY 2.0
"photo" by Lentini CC-BY2.0
"IMG_OZ02 (2 of 14)" by Kerry Goodwin Photography CC-BY 2.0
"Archive Video /: Party my tummy." by danoxster 2-BY-SA20
"Untitled" by Jacob Davies CC-BY-SA 2.0
Figure
"Steve Bobber finished, test ride Video by REDMAXSPIEDSSIOPCOM CC-BY-SA 2.0
"Natural Resources Program by Kentuckyguard
CC-BY 2.0
"Ray and Nana" by spilltojill CC-BY-SA 2.0
"HDV 0083"" by winsors CC-BY
"Somewhere Over the Ramnbow by Mikol CC-BY-SA 2.0
"IMG 2797" by Lentini CC-BY 2.0
Figure
"MOV(41 80" by mike_troiang CC-BY 2.0
"Vidco of hum playing drums" superhua
CC-BY220
"Day 08 Killarney' by brandonzeman
CC-BY-SA 2.0
"Jenner" by Mr. Gunn CC-BY-SA 2.0
"Ray 6 months" by spilltojill CC-BY-SA 2.0
"Hedwig's Theme" by Mikol CC-BY-SA 2.0
"5-20 (6)" by nmuster
CC-BY 2.0
Figure 7
"Bretagne" by MadMonday CC-BY 2.0
"Untitled" by Mack-Sama CC-BY-SA 2.0
"Natural Resources Program" by Kentuckyguard CC-BY 2.0
"Archive Video Party in my tummy." by danoxster CC-BY-SA 2.0
"Video Micah singing Old MacDonald" by superhua CC-BY 2.0
27


Reviews:
[{"review_id": "jnnQE1UDxes", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Pros**\n\nAudio-visual sound source separation is an impotant task. The paper pushes the boundary from specific domains (e.g. speakers, musics, etc) to generalized open-domain, which is crucial and far from trivial.\n\nThe authors introduced a new, large-scale, open-domain dataset for on-screen audio-visual separation. The videos span 2500 hours, 55 of which are verified by human labelers. The dataset will definitely be very useful for the community as it is way more diverse than before.\n\n---\n**Cons**\n\n*Related work*\n\nTo my understanding, Owens and Efros (2018) did not assume fixed number of speakers. While they validate their method under such setting, there is actually no limitation in their model that prevents them to have multiple on-screen sources. Therefore, I'm not sure about the first contribution, except the \"open-domain\" part. \n\n*Model*\n\nIn terms model architecture, (maybe I have missed something but) I didn't see much novelty in the current state. To me, the proposed model is simply a composition of multiple exisiting modules from previous work. Please note that I'm not saying building upon the sucess of previous efforts are wrong by any means. I just had the feeling that the authors are piecing various building blocks together w/o providing much intuition. Maybe there is some novelty lying within the current design. For instance, the authors may have developed a novel routing/module drawing inspiration from a certain observation; the combination of xxx and yyy is based on deliberate choice. It is, however, not clear to me at this point, at least the writing does not reflect it.\n\nFurthermore, if the network is the key player in this paper, the authors shall provide more evidence. While the authors do show conduct some ablation studies on the losses and data, there aren't any analysis regarding the importance of each module (e.g. how critical is the attention design?). It is thus difficult for readers to understand which part of the network is crucial for the success, and what is the major novelty within the architecture. The current form provides very litte intuition and take away.\n\n*Objectives*\n\nEq. 4 and Eq. 6 looks very similar to me. Aren't they equivalent if the **A** in Eq. 6 is the same as **A** that minimizes Eq. 2, since the assignment in Eq. 4 comes from Eq. 2? On the other side, if the two **A** are different, what's the intuition of exploiting different A for different loss?\n\n\n*Writing/Presentation*\n\nThe flow of the model architecture section can be improved. The authors did not provide any high-level context. Instead, they simply dig into the \" details\" of each module. I'm not aware of the connections among while reading the text. Instead, I need to constantly check the figure and infer these.\n\nI also don't know what is the input/output of the model and what representations they are using. Shoudn't these be explained at the very beginning? These are not explicitly defined and I need to infer them myself. For instance, is the output of masking network a M x T soft mask with values lying within [0, 1]? do they exploit waveform (fig. 2) or spectrogram (fig. 1) for audio? I figured/inferred a lot these out after I moved to the experiment section. But from two cents, these are related to the model.\n\n*Experiments*\n\nCurrently the authors only evaluate the model on their own dataset. How does the model work on existing datasets? For instance, AudioSet, MUSIC, FAIR-Play, etc. It seems that there is nothing preventing them from applying their model to those datasets. Without these results, it would be hard to justify if the proposed \"open-domain model\" can generalize to \"a specific domain.\" I think at least  direct inference (generalization) or train from scratch is required. \n\nFurthremore, the authors did not compare with any baseline. It seems to me that quite a few prior art [Owens and Efros (2018), Hang et al (2018), etc] can serve as baselines with minor modification. Take Owens and Efros (2018) for example. While they may not be able to decompose each sound source within the on-screen mixture, one can still leverage it to evaluate the on/off-screen separation. The authors thus shall be able to report SI-SNR too. Otherwise its very difficult for people to do an apple-to-apple comparison of this work and prior work.\n\nThe authors should report more performance at more percentiles. The most illustrative way is to show the cumulative plot - how many % of data have error less than x.\n\nIs there an intuition of why only pre-training part of the model? Why not pre-train the separation network too?\n\n---\n**Minor comments**\n\nHow do the authors define the diversity (Sec. 5.1) of the videos? Do they make use of the tags provided by YFCC100M? Also, whats the statistics of those filtered data? Would be great to provide more details so that we know its indeed covering a wide range of semantic categories.\n\nSome relevant literatures are missing. For instance, [1] also associates the visual information with speeching signal. The subjectives (eg person, dog, birds) in the paper can be viewed as on-screen audio, while prepositions can be seen as off-screen voice. There are definitely a lot more on this direction, but this paper pop out my head right away.\n\n[1] Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input. ECCV 2018.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SGqi9o6sY4P", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an unsupervised method for open-domain, audio-visual separation system. The proposed model was optimized using the newly suggested mixture invariant training (MixIT) together with a cross entropy loss function. The authors suggest to separately process the audio and video, and next align them with a spatiotemporal attention module.\n\nUnsupervised source separation, especially for open-domain is an interesting and important research direction. However, there are several concerns with this submission that need to be addressed first.\n\nMy main concern is the contribution of this paper. The authors presented a fairly complicated system comprised of several modules. I would expect the authors to run an ablation study / analysis to better understand their contribution to the final model performance. For instance, why do we need attentional pooling? do we need it in both audio and video? When does the model fail? can we learn something from it? \n\nSecond, I know the authors said this is the first system to do so, however, I would still expect the authors to compare to some baseline. Maybe a fully supervised one? Otherwise it is hard to interpret the numbers presented in Table 1. It is hard to understand how good is this system and how much room for improvement do we have. \n\nRegarding the samples, it is a bit hard to interpret this results. For every file there are 5 videos and 5 separated samples, some of them sound almost identical. Again, there is no baseline to compare against, so it is hard to understand how good the quality of the separations is. I suggest the authors to improve the samples page to better present emphasis their results. \n\nA question for the authors, since you treated this task at an unsupervised task, did you try to run some subjective evaluations? maybe let users annotate the sound files and compare variance?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Aker66GGh-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThis paper proposes a multi-modal sound source separation framework in which they aim to separate on-screen sound. The proposed method extends the recent unsupervised source separation framework MixIT by conditioning video input. Although there have been numerous multi-modal sound source separation papers, this work goes one step further by using the sound data “in-the-wild”. The experiment results show reasonable performance on separating mixture from mixtures.\n\n\n\nStrength: \n\nThis paper extends the existing source separation approaches to enable multi-modal source separation.\n\n\n\nWeakness:\n\n-----Novelty-----\n\nThe proposed method is a combination of existing approaches, which shows mere novelty.\n\n-----Writings-----\n\nSome unexplained notations and typos.\nIf the notation appears for the first time in the paper the authors must explain what it is.\nFor example,  \n\n1. In equation (6) please explain the notation \"P(?)\" in the following sentence: \"minimum loss over all settings P(R) of the labels\"\n\n2. In section 4.1 what does Ms stand for? \n\"We use a MixIT separation loss (Wisdom et al., 2020b), which optimizes the assignment of M\nestimated sources ^s = Ms (x1 + x2) to two reference mixtures x1, x2 as follows\"\n\nSome sections (especially methods) are poorly explained.\nFor example,\n“following the same procedure as Tzinis et al. (2020)” -> following how?\n“The concatenated visual embedding is resampled, fed through a dense layer, and concatenated together with all convolutional blocks.” -> resampled how? What resampling method did you use?\n\nFurthermore, I’d like to ask the authors to specify exact neural architectures at least in Appendix.\n“We also use local features extracted from an intermediate level in the visual convolutional network, that has 8 × 8 spatial locations” -> from which layer did you take the local video embedding that has the size of 8x8?\n\n-----Experiments-----\n\nI understand that it is hard to obtain single source on-screen clips, but it could have been better if the authors had collected some small samples and test the single mixture separation performance.\n\n\nOverall:\n\nThe unsupervised separation framework is an important research direction to deal with the real-world sound sources. Although the novelty is mild, therefore, I think this work shows a promising research direction, hence recommend a weak accept.\n\n\nQuestions:\n\n1.\t5 video frames for 5-second of video seems too small to me. Are there any reasons for choosing such small number of frames for conditioning?\n\n2.\tWhy does Global video embedding have to be an input for On-screen classifier?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3EfpTIXdRNY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a system for separating \"on-screen\" sounds from \"off-screen\" sounds in an audio-visual task, meaning sounds that are associated with objects that are visible in a video versus not. It is trained to do this using mixture invariant training to separate synthetic mixtures of mixtures. It is evaluated on a subset of the YFCC100m that is annotated by human raters as to whether the clips have on-screen, off-screen, or both types of sounds, with the predictions of a previously described model  (Jansen et al, 2020) helping to reduce the number with only off-screen sounds. The predictions are evaluated in terms of how well they can estimate the true on-screen sound (in terms of SI-SNR) and how well they can reject off-screen sound (in terms of a metric called off-screen suppression ratio, OSR). The results show that the system can successfully distinguish between on- and off-screen sound, but that different training regimens lead to different tradeoffs in these two metrics. The system with the best SI-SNR (8.0 dB) is trained using just data from the previous model along with the mixture invariant training criterion.\n\nThe paper presents an interesting approach to solving the on-screen vs off-screen sound problem in audio-visual source separation. While other approaches have solved similar problems for more specific source types (speech, music), this one does appear more \"universal\", with few assumptions tying it to a specific sound type. While this novelty is one of the strengths of the paper, it makes it more difficult to evaluate the system in comparison to established baselines. Such baselines would make it easier to understand how well the system is doing and which parts of it are the most useful and important. Perhaps it could be compared to one of the more source-specific systems on test data suited to such a system.  Another useful benchmark would be using the audio-visual coincidence prediction system of Jansen et al (2020) to assign the entire soundtrack to on or off screen and measuring the various evaluation metrics on the predictions (although infinities in the metrics make this tricky). Yet another baseline might be using an audio-only mixture of mixtures separation system, perhaps with an oracle assignment system. There is a baseline that uses oracle mixtures to compute an estimate of the relevant signals, but the opposite would also be informative as to the utility of the visual component of the system and problem. \n\nWhile these additional baselines would be nice, I do not think that they are necessary for publication presently. The experiments reported are conducted thoroughly and carefully. The results, while leaving certain aspects of the optimal training program underspecified, demonstrate that the system is useful. \n\nOne additional aspect of the system that could be quantified more fully is the quality of the individual source separations. While the on- vs off-screen task is evaluated thoroughly, there is no quantitative evaluation of the source separation performance for individual sources in each mixture. It would be possible to evaluate on completely synthetic mixtures, although this is perhaps outside of the main contribution of the paper, the audio-visual combination.\n\nOverall, this is an interesting approach that is well described. The evaluation is sufficient for the on- vs off-screen task, although not sufficient to judge whether the system has learned a completely unsupervised source separator, making the scope of the contribution somewhat more limited than it has the potential to be.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new, large-scale, open-domain dataset for on-screen audio-visual separation, and provides an initial solution to this task. As the setting is quite specialized, the authors proposed a neural architecture based on spatial-temporal attentions (while using existing learning objective for audio separation). The reviewers were initially concerned that, while reasonably motivated, the architecture seemed some arbitrary. The authors then provided extensive ablation studies to evaluate the significance of each component with existing datasets, and these efforts are appreciated by reviewers. The authors may consider re-organizing the paper and moving some ablation studies to the main text. On the other hand, the reviewers believe that the dataset will be very useful for the community due to its diversity in content and label quality.
Decision: Accept (Poster)