Deep Audio Prior

To understand the robustness of the deep audio prior , we construct a benchmark dataset Universal-150 for universal sound source separation with a diverse set of sources . We show superior audio results than previous work on both qualitative and quantitative evaluations . We also perform thorough ablation study to validate our design choices . https : //iclr-dap.github.io/Deep-Audio-Prior/ 1 INTRODUCTION . Typically , a deep neural network distills robust priors from a large amount of labeled or unlabeled data ( Deng et al. , 2009 ; Jansen et al. , 2018 ) . In audio research , neural networks such as VGGish ( Hershey et al. , 2017 ) and Wave-U-Net ( Stoller et al. , 2018 ) have shown great success in audio classification , audio source separation , and many other challenging tasks . While large audio datasets have greatly improved supervised training , the collection and especially the cleaning of a large amount of audio data still remain an open challenge ( Fonseca et al. , 2017 ) . For example , in AudioSet ( Gemmeke et al. , 2017 ) , one of the popular audio datasets , we often find audios/videos that contain content beyond what the label specifies1 . In this paper , we combine the power of deep neural networks and temporal prior in audio without any external training data . Similar ideas have been explored in deep image prior ( DIP ) ( Ulyanov et al. , 2018 ) . Double-DIP from Gandelsman et al . ( 2019 ) further showed that it is possible to achieve robust unsupervised image decomposition from a single-image input , without pre-training on any data . Deep Audio Prior ( DAP ) ’ s capability to train on a single audio file has several advantages . First , with proper selection of the audio priors , we show that DAP generalizes well to a wide variety of unseen types of data . Second , our training process is fully unsupervised and therefore make it possible to pre-process large volumes of data in the wild . Last but not least , we show several novel applications that are only possible because of the unique features of DAP , including universal source separation , interactive editing , audio texture synthesis , and audio co-separation . Coherence and discontinuity in audio . Domain gap between audio and visual images precludes direct adoption of the image priors . Many assumptions or priors that are true for images no longer hold for audio . By nature , audio signals exhibit strong temporal coherence , e.g . one ’ s voice changes smoothly ( See A→ B in inset ) . Since images tend to have more spatial patterns , most existing deep image priors have focused on how to encapsulate the spatial redundancy . Another challenge specific to audio is the activation discontinuity . Unlike in videos where an object moves continuously in the scene , a sound source can make sound or turn complete silence at any given time ( see A→ C in inset ) . Our proposed deep audio prior framework has the following main contributions : • A temporally coherent source generator that can reproduce a wide spectrum of natural sound sources including human speech , musical instruments , animal sounds , etc . 1As an example , under “ bark ” from AudioSet , we find https : //youtu.be/2mJbGx5D-zA ? t=150 containing human speech , wind noises , and various audio effects other than bark . • A novel mask activation scheme to enable and disable sources with frequency binning and without temporal dependence . • We demonstrated the effectiveness of DAP in several challenging applications , including universal audio source separation , interactive mask-based editing , audio texture synthesis , and audio co-separation . • We also introduce Universal-150 , a benchmark dataset on blind source separation for universal sound sources . We showed that DAP outperforms other blind source separation ( BSS ) methods in multiple numerical metrics . 2 DEEP AUDIO PRIOR FRAMEWORK . Deep Audio Prior is an unsupervised blind source separation framework . More specifically , DAP does not train on any extra data other than the input audio mixture . Similar to image foreground and background segmentation , audio blind two-source separation can be expressed as : Smixture = S1 ( z1 ) ∗M1 ( g1 ) + S2 ( z2 ) ∗M2 ( g1 ) ( 1 ) where S1 and S2 are two audio generator networks , M1 and M2 are two mask networks , z1 , z2 and g1 , g2 are sampled from random distributions . Our method works in Time-Frequency ( T-F ) spectrogram space ( Cohen , 1995 ) . All the input and output variables are of dimension F × T . Waveforms are transformed into spectrograms by Short-Time Fourier Transform ( STFT ) with a frame size of 1022 and hop size of 172 , and audio sampling rate is 11000 Hz . Figure 1 shows an overview of our framework . The intuition for our single-audio source separation is inspired by single-image decomposition : it is much easier for two generator networks ( S1 and S2 ) to learn two distinct sound sources respectively , rather than forcing one of the generators to learn the mixture . Gandelsman et al . ( 2019 ) analyzed this in terms of complex mixture versus simple individual components . In audio , as shown in §3.1 , we show that networks tend to quickly learn patterns from two distinct sources . Reconstruction Loss Let S∗1 = S1 ( z1 ) ∗ M1 ( g1 ) and S∗2 = S2 ( z2 ) ∗ M2 ( g2 ) , then we have S∗mixture = S ∗ 1 + S ∗ 2 . Clearly , when combining the separated sounds S ∗ 1 and S ∗ 2 , we should obtain the original sound mixture Smixture . The data fidelity term is expressed as a reconstruction loss , lrec , which will push the combination of the two separated sounds to be close to the original sound mixture . lrec = ||Smixture − S∗1 − S∗2 ||2 . ( 2 ) 2.1 TEMPORALLY COHERENT AND DYNAMIC SOURCE GENERATOR . Temporally Coherent Source Generation Sound signals from the same source across time would be similar ( Rosen , 1992 ) . To explicitly model the temporal property , we use multiple audio frames with temporal consistent noise as inputs for each sound source . We split the Smixture into N frames along the time axis and obtain { Simixture } Ni=1 . Accordingly , we use n noise input pairs { zi1 , zi2 } Ni=1 to predict the corresponding sounds { S1 ( zi1 ) , S2 ( zi2 ) } Ni=1 . For the rest of this paper , we will omit the underscript when there is no confusion , i.e . zi instead of zi1 . To explicitly enforce the temporal coherence , we impose strong correlations on input noises : z1 = n∗ and zi = zi−1 + ∆ui ( 3 ) where ∆ui is a random noise sampled from an uniform distribution with a variance significantly lower than that of Gaussian noise n∗ initialization of z1 . Since we use shared networks S1 and S2 across different frames for predicting individual sounds and the noise inputs are temporally consistent , which will enforce the network to predict temporal consistent sounds . A similar idea was also adopted in ( Gandelsman et al. , 2019 ) to preserve video coherence but they use the noise to predict masks . We also employ a temporal continuity loss function to further enforce it by pushing the absolute of gradients along time to be small ( Chambolle , 2004 ) : ltc = ∑ i ∑ p ∑ q |Si ( p , q ) − Si ( p , q − 1 ) | . ( 4 ) Dynamic Source Generation Audio signals can also have dynamic patterns with large variations . Since values in zi− zi−1 = ∆ui are very small by construction , the temporally consistent noise can only handle small variations and is not capable of capturing temporally dynamic sound patterns . To preserve the temporally consistent patterns in predictions and also hallucinate dynamic patterns , in spirit of curriculum learning ( Bengio et al. , 2009 ) , we gradually add dynamic noise into inputs as we progress more training iterations : zi = α ( t ) zi−1 + ∆ui + ( 1− α ( t ) ) ni ( 5 ) where the t denotes optimization iteration , and ni refers to a random Gaussian noise . Unlike the temporal consistent noise in ( 3 ) that have a constant n∗ as initialization , ni is independently sampled for each frame i . To balance the n∗ and ni throughout our training iterations , we introduce a coefficient α ( t ) : α ( t ) = 1 if t < T1 · · · initially zi are very smooth , T2−t T2 if T1 ≤ t ≤ T2 · · · gradually transit from smooth to dynamic inputs , 0 if t > T2 · · · at the end zi are sufficiently dynamic , ( 6 ) ! '' 1 0 ! % & ' ( & ) Sharp change in red better captures dynamic variation . where T1 and T2 are two thresholds and T2 > T1 . If t < T1 , only a temporally shared constant noise n∗ is used ; if T1 ≤ t < T2 , the dynamic sampled noise ni will be gradually added ; if t > T2 , only a dynamic noise is used . The noise input design will first let the model predict stable temporally consistent sound patterns and then push the model to capture large variations in spectrograms to reduce the reconstruction loss . Note that α ( t ) is not continuous with respect to t ( see red curve in inset ) . In practice , we found this abrupt change in α ( t ) improves the ability to capture dynamic audio change . See more discussion in §4 . Frequency Domain Exclusion We assume that two different sounds S∗1 and S∗2 are dissimilar . To enforce the constraint , we utilize an exclusion loss function from Zhang et al . ( 2018 ) in which it is formulated as the product of normalized gradient fields of the two sound predictions at L spatial resolutions : lex = L∑ n=1 ||Ψ ( S∗ ( n ) 1 , S ∗ ( n ) 2 ) ||F and Ψ ( x , y ) = tanh ( λ1|∇x| ) tanh ( λ2|∇y| ) , ( 7 ) where λ1 and λ2 are regularizers , n the spatial patch index , || · ||F the Frobenius norm , denotes element-wise multiplication . We set L = 3 , λ1 = √ ||∇y||F ||∇x||F , and λ2 = √ ||∇x||F ||∇y||F . 2.2 1D MASK CONSTRAINTS FOR AUDIO . Sound sources will not always make sounds all the time , which would break the temporal consistency in the spectrogram domain . To address this issue , we introduce audio masks to activate sounding spectrum regions and deactivate silent regions . If a source is sounding at a time , the spectrum bin at the time should be activated . Based on the observation , mask values within the same temporal bin should be consistent and binary . Namely , if a dog barking sound is present , it should appear across all frequency range under the same timestamp . Therefore , we force the mask within the same temporal bin to be consistent : Mi ( 0 , q ) = Mi ( 1 , q ) = ... = Mi ( F − 1 , q ) = mqi , where i refers to i-th sound source , q ∈ { 0 , 1 ... , T } , F and T are height ( frequency axis ) and width ( time axis ) of the spectrogram M . In our implementation , we use a max pooling operator to aggregate the mask content along frequency-coordinate and generate mask value mqi using a Sigmoid function . m q i represents the mask for i-th source at time q . We define m q i to be a continuous variable for ease of optimization and will enforce an extra binary loss term . Non-Zero Masks Furthermore , at any time , if there is a sound in Smixture , at least one of the masks should be activated . To this end , we introduce a nonzero mask loss term : lnonzero = ∑ q wq · ( + min ( σ , ∑ i mqi ) ) −1 and wq = ∑ p log ( 1 + Smixture ( p , q ) ) , ( 8 ) where = 10−6 for numerical stability and σ = 1 is a margin value . The margin value is to ensure when the sum of mask activation is already larger than the σ , the loss will not continue to push the mask values . wq is to suppress this loss term if there is no sound in Smixture at time q . Moreover , in our observations , we found that masks are either fully activated or fully deactivated . Very rarely would a sound source be at 50 % activation level . Therefore , we introduce a differential loss term to encourage the mask networks to generate binary masks : lbinary = ∑ i ( + ∑ p , q |Mi ( p , q ) − 0.5| ) −1 , where again = 10−6 is used to avoid numerical issues . The lbinary will force the mask values to be as far away as possible to 0.5 , which means they are close to either 0 or 1 .

Summary: The authors detail a set of priors for unsupervised decomposition of individual spectrograms into their component parts. The introduce reasonable constraints on temporal coherence (consistency and dynamic shifts) and mask activations (at least one component always activated). They also regularize sources to not overlap spectrotemporally. Decomposition is performed by training the weights of a U-Net on a single spectrogram as in deep image priors. The authors demonstrate quantitative improvements on blind source separation over other data-agnostic techniques, and qualitative use of the model for interactive editing, audio texture synthesis, and audio watermark removal. The work also performs an ablation study to qualitatively demonstrate the importance of each element for the prior. The experiments are performed well and explained clearly. They also introduce a dataset of diverse mixtures for future comparisons.