Deep Audio Prior
Authors: Yapeng Tian
Chenliang Xu
Dingzeyu Li
Venue:  (2020)
Keywords: deep audio prior
blind sound separation
deep learning
audio representation

Abstract: Deep convolutional  neural networks are known to specialize in distilling compact and robust prior from a large amount of data. We are interested in applying deep networks in the absence of training dataset. In this paper, we introduce deep audio prior (DAP)  which leverages the structure of a network and the temporal information in a single audio file. Specifically, we demonstrate that a randomly-initialized neural network can be used with carefully designed audio prior to tackle challenging audio problems such as universal blind source separation, interactive audio editing, audio texture synthesis, and audio co-separation.

To understand the robustness of the deep audio prior, we construct a benchmark dataset Universal-150 for universal sound source separation with a diverse set of sources. We show superior audio results than previous work on both qualitatively and quantitative evaluations. We also perform thorough ablation study to validate our design choices.

Under review as a conference paper at ICLR 2020
DEEP AUDIO PRIOR
Anonymous authors Paper under double-blind review
ABSTRACT
Deep convolutional neural networks are known to specialize in distilling compact and robust prior from a large amount of data. We are interested in applying deep networks in the absence of training dataset. In this paper, we introduce deep audio prior (DAP) which leverages the structure of a network and the temporal information a single audio file. Specifically, we demonstrate that a randomly-initializer neural network can be used with carefully designed audio prior to tackle challenging audio problems such as universal blind source separation, interactive audio editing, audio texture synthesis, and audio co-separation.
To understand the robustness of the deep audio prior, we construct a benchmark dataset Universal-150 for universal sound source separation with a diverse set of sources. We show superior audio results than previous work on both qualitative and quantitative evaluations. We also perform thorough ablation study validate our design choices. https://iccr dap. github. io/Deep Audio-Prion Prior/
INTRODUCTION
Typically, deep neural network distills robust priors from large amount of labeled unlabeled data (Deng et al. 2009, Jansen et al.) 2018). In audio research, neural networks such as VGGish (Hershey al., 2017) and Wave-U-Ne U-Net (Stoller et al., 2018) have shown great success in audio classification, audio source separation, and many other challenging tasks. While large audio datasets have greatly improved supervised training, the collection and especially the cleaning of a large amount of audio data still remain an open challenge (Fonseca et al.] 2017). For example, in AudioSet (Gemmeke et al. 2017), one of the popular audio datasets, we often find audios/videos that contain content beyond what the label specifies"
In this paper, we combine the power of deep neural networks and temporal prior in audio without any external training data. Similar ideas have been explored in deep image prior (DIP) (Ulyanov et al.) 2018). Double-DIP from Gandelsman al. (2019) further showed that it is possible to achieve robust unsupervised image decomposition from single-image input, without pre-training on any data. Deep Audio Prior (DAP)'s capability to train on a single audio file has several advantages. First, with proper selection of the audio priors, show that DAP generalizes well to a wide variety of unseen types of data. Second, our training process is fully unsupervised and therefore make possible to pre-process large volumes of data in the wild. Last but not least, show several novel applications that are only possible because of the unique features of DAP, including universal source separation, interactive editing, audio texture synthesis, and audio co-separation.
Domain gap between audio and visual images precludes direct adoption of the image priors. Many assumptions priors that true for images no longer hold for audio. By nature, audio signals exhibit strong temporal coherence, e.g. one's voice changes smoothly (See A + B in inset). Since images tend to have more spatial patterns, most existing deep image priors have focused on how to encapsulate the spatial redundancy. Another challenge specific to audio is the
activation discontinuity. Unlike in videos where an object moves continuously in the scene, a sound source can make sound or turn complete silence at any given time (see A + C in inset).
A B C
Coherence and discontinuity in audio.
Our proposed deep audio prior framework has the following main contributions:
A temporally coherent source generator that can reproduce a wide spectrum of natural sound sources including human speech, musical instruments, animal sounds, etc.
's an example, under "bark" from AudioSet, we find https: /tt///yyttuuu be/pm//mx/ii zA?t- 150 containing human speech, wind noises, and various audio effects other than bark.

---

Under review as a conference paper at ICLR 2020
21
91
M
mixture
Reconstructed
M2
DAP framework illustrated with synthetic sound mixture spectrogram. With random noises as inputs, we use two sound prediction networks: S and S2 and two mask modulation networks: M1 and M2 perform source separation. The four networks share same U-Net structure (Ulyanov et al. 2018).
Figure 1:
A novel mask activation scheme to enable and disable sources with frequency binning and without temporal dependence.
We demonstrated the effectiveness of DAP in several challenging applications, including universal audio source separation, interactive mask-based editing, audio texture synthesis, and audio co-separation.
We also introduce Universal-150, benchmark dataset blind source separation for universal sound sources. We showed that DAP outperforms other blind source separation (BSS) methods in multiple numerical metries.
DEEP AUDIO PRIOR FRAMEWORK
Deep Audio Prior is an unsupervised blind source separation framework. More specifically, DAP does not train on any extra data other than the input audio mixture. Similar to image foreground and background segmentation, audio blind two-source separation can be expressed as:
Smixture = (11) M1 (91) + S2(22) M2(91)
(1)
where S1 and S2 are two audio generator networks, Mj and M2 are two mask networks, 21, 22 and 91,93 are sampled from random distributions. Our method works in Time-Frequency (T-F) spectrogram space (Cohen) [995). All the input and output variables are of dimension x
Waveforms transformed into spectrograms by Short-Time Fourier Transform (STFT) with
frame size of 1022 and hop size of 172, and audio sampling rate is 11000 Hz. Figure shows an overview of our framework.
The intuition for our single-audio source separation inspired by single-image decomposition: is much easier for two generator networks (S, and 52) to learn two distinct sound sources respectively, rather than forcing one of the generators to learn the mixture. Gandelsman et al. (2019) analyzed this in terms of complex mixture versus simple individual components. In audio, shown in $3.1, we show that networks tend to quickly learn patterns from two distinct sources.
Reconstruction Loss Let Si Si(21) Mi(gi) and S S2(22) M2(g2), then have S Smixture = St + Sz. Clearly, when combining the separated sounds S and S we should obtain the original sound mixture Smixure- The data fidelity term is expressed as a reconstruction loss, Inc. which will push the combination of the two separated sounds to be close to the original sound mixture.
1.e
Smixxtue S --2112
2

---

Under review as a conference paper at ICLR 2020
2.1 TEMPORALLY COHERENT AND DYNAMIC SOURCE GENERATOR
Temporally Coherent Source Generation Sound signals from the same source across time would be similar (Rosen 1992). To explicitly model the temporal property, We use multiple audio frames with temporal consistent noise as inputs for each sound source. We split the mmixture into N frames along the time axis and obtain mixxuuee [==- Accordingly, we use noise input pairs {zi, z 22f==1 to predict the corresponding sounds (21), {> For the rest of this paper, we will omit the underscript when there is no confusion, i.c. 2² instead of z1.
1.N
To explicitly enforce the temporal coherence, impose strong correlations input noises:
= n and 2² = + Au'
(3)
where Aut is a random noise sampled from an uniform distribution with a variance significantly lower than that of Gaussian noise n' initialization of 2 Since we use shared networks Si and S2 across different frames for predicting individual sounds and the noise inputs are temporally consistent, which will enforce the network to predict temporal consistent sounds. A similar idea was also adopted in (Gandelsman et al. 2019) to preserve video coherence but they use the noise to predict masks. We also employ temporal continuity loss function to further enforce it by pushing the absolute of gradients along time to be small (Chambolle 2004
he = EΣΣΣ (.(p.g) S, 1)/.
(4)
Dynamic Source Generation Audio signals can also have dynamic patterns with large variations. Since values in 2¹ - 2f-1 Aut are very small by construction, the temporally consistent noise can only handle small variations and is not capable of capturing temporally dynamic sound patterns. To preserve the temporally consistent patterns in predictions and also hallucinate dynamic patterns, in spirit of curriculum learning (Bengio et al. 2009), we gradually add dynamic noise into inputs as progress more training iterations:
= =a= Au' +11 - a(t))
(5)
where the denotes optimization iteration, and n² refers to a random Gaussian noise. Unlike the temporal consistent noise in (3) that have constant n' as initialization, n' is independently sampled for each frame To balance the n² and n² throughout our training iterations, we introduce coefficient (t):
< if ST
initially z' are very smooth, gradually transit from smooth to dynamic inputs, at the end zf are sufficiently dynamic,
att) =
Ta-t T2
(6)
0
i T2
where Ti and T2 are two thresholds and T3 >> T1. If < Tı, only temporally shared constant noise n² is used; T; < t < Tg. the dynamic sampled noise n² will be gradually added; if > T2, only a dynamic noise is used. The noise input design will first let the model predict stable temporally consistent sound patterns and then push the model to capture large variations in spectrograms to reduce the reconstruction loss. Note that a(I) is not continuous with respect to / (see red curve in inset). In practice, found this abrupt change in (l) improves the ability to capture dynamic audio change. See more discussion in $4.
a(t)
1
0
T
T2
Sharp change red hetter captures dynamic variation.
Frequency Domain Exclusion We assume that two different sounds Sj and St are dissimilar. To enforce the constraint, we utilize an exclusion loss function from Zhang et al. (2018) in which it formulated as the product of normalized gradient fields of the two sound predictions at L spatial resolutions:
la Σ (()(()))) ("))) and (xx,)) tanh (z) @ tanh
(y2)
where X and d are regularizers, n the spatial patch index, || the Frobenius norm, @ denotes element-wiss multiplication. We set L = 3, λ1
V
and X = Vy
3

---

Under review as a conference paper at ICLR 2020
2.2 ID MASK CONSTRAINTS FOR AUDIO
Sound sources will not always make sounds all the time, which would break the temporal consistency in the spectrogram domain. To address this issue, we introduce audio masks to activate sounding spectrum regions and deactivate silent regions.
If source is sounding at time, the spectrum bin at the time should be activated. Based on the observation, mask values within the same temporal bin should be consistent and binary. Namely, if dog barking sound is present, it should appear across all frequency range under the same timestamp. Therefore, we force the mask within the same temporal bin to be consistent: M0 q) = M, =
M 1, g) m. where i refers to i-th sound source, q {0, 1..., T) F and T are height (frequency axis) and width (time axis) of the spectrogram M. In our implementation, we use a max pooling operator aggregate the mask content along crequency-coorddnate and generate mask value m. using a Sigmoid function. m. represents the mask for i-th source at time q. We define my to be a continuous variable for ease of optimization and will enforce an extra binary loss term.
Non-Zero Masks Furthermore, at any time, if there is sound in Smixure, at least one of the masks should be activated. To this end, we introduce a nonzero mask loss term:
Inonzero Σ Wgg (E + min(o, Σ Em?))= and Wi Σ log(1 + Smixture (p, q)),
(8)
where 10-6 for numerical stability and = 1 is margin value. The margin value is to ensure when the sum of mask activation is already larger than the a, the loss will not continue to push the mask values. 4 is to suppress this loss term if there is no sound in Smixtue time q. Moreover, in our observations, we found that masks are either fully activated or fully deactivated. Very rarely would a sound source be 50% activation level. Therefore, encourage the mask networks to generate binary masks: /pinary = E.( +E [P v) 0.5))-1, where again 10-6 is used to avoid numerical issues. The Iinaary will force the mask values to be as far away as possible to 0.5, which mcans they are close to either 0 or 1.
introduce differential loss term to
2.3 WALKTHROUGH ON SYNTHETIC SEPARATION EXAMPLES
With all the pieces together, our total loss is defined as:
Luttal = Loee + lic + Lex + Inomzero + Ieinary-
We optimize the networks in end-to-end manner with all the loss functions. We empirically keep the weights for all loss terms the same, with the only exception being loinary has a 0.01 factor. To validate the implementation of our algorithm, generate two types of synthetic spectrograms.
(i) Single-frequency band fake sounds We generate a synthetic mixture where each of the two audio sources is producing flat tone single frequency. For this test, since we know the spectrogram at different segment will always produce the same output (each sound source just =f for sound source 1 and the same for source 2.
flat bar), set the input noise 2t 2f The top two rows from Figure 2 show that our DAP achieves perfect prediction on both the source generators and masks.
(a) Input Mixture
(b) Predicted S₁ and S2 (c) Predicted M, and M2
(d) Final Si and Sž
Figure Synthetic tests: two single-tone sources (top two rows) and two cosine-tone sources (bottom two rows). Given single input mixture, DAP achieves perfect separation on both inputs.
4

---

Under review as a conference paper at ICLR 2020
(a) Input Mixture
(b) NMF
(c) RPCA
(d) KAM
(c) DAP
(f) Groundtruth
Figure 3: Audio comparison on Universal- 150 benchmark. Qualitatively, DAP significantly outperforms all the other blind separation methods. For full results, please check out our project page: ht hhttps://iclr-dapp..
io Deep-Aud-sdeee o
(ii) Curved input sounds Our next test is to validate our DAP framework can handle temporally coherent spectrogram changes. Two shifted cosine curves are combined together as input to our separation pipeline (see bottom two rows in Figure 2). We set the input noise according to equation (3). Again, DAP framework achieves the desired separation of sources and masks.
APPLICATIONS
Next, present several challenging applications with DAP. For the best experience, please use Adobe PDF Reader or visit our anonymous webpage to listen to our audio results. Unless specified otherwise, all the experiments for the same application are run with the same set of parameters.
3.1 UNIVERSAL BLIND SOURCE SEPARATION
Given sound mixture Smixture universal blind source separation aims separate individual sounds from the sound mixture without using any external data. Universal blind separation is challenging because the input audio can be in arbitrary domain, not just commonly studied speech or music domains.
Universal-150 Audio Benchmark To the best of our knowledge, there is no publicly available universe audio source separation dataset. Therefore, we built such dataset that contains 150 audio mixtures and each sample is two-sound mixture. These mixture samples come from pairs of 30 unique sounds from YouTube and ESC50 sound classification dataset (Piczak 2015) covering large range of sound categories appeared in our daily life, such as animal (e.g., dog, cat, and rooster), human (e.g., human speech, baby crying, and baby laughing), music (e.g., violin and guitar), natural sounds (e.g.. rain, wave, and crackling fire), domestic and urban sounds (e.g., clock, keyboard typing, and siren).
Comparison with Blind Separation Methods We compare our method with several blind source separation (BSS) methods: non-negative matrix factorization (NMF) from Spiertz & Gnann (2009), robust principal component analysis (RPCA) from Huang et al. (012)), and kernel additive modelling (KAM) from Yela et al. (2018). Figure 3 shows that our method outperforms the compared methods qualitatively. Please either use Adobe PDF Reader and click on each spectrogram visit webpage to listen to the audios. For implementations, use (Manilww al. 2018 for NMF and RPCA and (Yela et al. 2018) for KAM.
IIn Adohe PDF Reader, click on each spectrogram image and enable the audio plugin when prompted. *uur anonymous submission page: https://cllr-appgttr thub. to/Deep-Aree Audi Prior/
5

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison between DAPINMERPCA/KAM on numerical metries: SDR/SIR/LSD, For SDR/SIR, higher is better and for LSD, smaller is better. The best results on the three metrics are from our method, DAP.
NMF
RPCA
KAM Yela al. 2018
Proposed DAP
Spiertz& & Gnann 2009
(Huang et al. 2012]
SDR
-6.369
-5.395
-3.375
-1.581
SIR
-1.934
-0.227
-0.753
3.859
LSD
2.301
2.011
2.321
1.959
Moreover, to quantitatively evaluate sound separation performance of different BSS methods, we run separation on all 150 sounds and compare these methods in three metrics: hignal-to-Distoriin Ratio (SDR), Signal-to- Intrrffeeenee Ratio (SIR), and Audio Spectrum Distance (LSD) (Vincent et al. 2006; Morgado et al. 2018). The SDR and SIR measure the distortion and interferences in separated audios. L.S measures the Euclidean distance between predicted audio spectrum magnitude and the corresponding ground truth audio spectrogram magnitude.
DAP outperforms these three BSS methods in all three numerical metrics, as shown in Table 1. Note that all these four methods, including DAP, do not require any training data. The only input to the algorithm is the single mixture audio file.
Comparison with Methods using Deep Networks Since existing deep audio separation networks were usually trained on music or speech data, they are supervised and can not handle unseen sounds in our universe sound separation dataset. So we compare our method with two state-of-the-ar deep audio networks: Deep Network Prior (DNP) from Michelashvili & Wolf (2019) and Speech Enhancement Generative Adversarial Network (SEGAN) from Pascual et al. (2017] 7) on noisy speech data. Figure 4 illustrates speech denoising results. For supervised model, in tests, SEGAN works well on noises that are similar to the training set. However, for unseen novel noises like the keyboard typing noise in Figure (a), SEGAN did not remove the noise. The proposed DAP can remove background noise. A side effect of DAP aggressively removing noises is the excessive removal of speech signals, which might lead to lower quality in some cases. Note that we do not impose any speech-related denoising prior in the universal DAP model.
3.2 INTERACTIVE MASK-BASED EDITING
Since the output of our method contains both mask and the sound generator, we can add additional constraints on either the mask or the sound spectrogram. Traditionally this type of interaction is only performed on spectrograms since they don't 't' have the predicted masks available (Bryan et al, 2014). There are several brrabacks with using spectrogram strokes: spectrograms are non-intuitive for aud-o professionals, and even then, lot of strokes needed for real-world audios.
In comparison, constraints on ID masks are a lot easier for users to specify. Figure shows the simple ID box given by the user can quickly improve the results. These ID box constraints can be easily drawn by users that are not familiar with frequency spectrograms. Basically they select
(a) Noisy Speech
(b) DNP
(c) SEGAN
(d) DAP
Figure 4: Comparison with other deep network models. DNP does not require training dataset and DAP achieve higher denoising quality. As for supervised models, for example SEGAN, that are trained on large dataset, although they usually work well on seen noises, it is hard to generalize to unseen noise. Here the speech is mixed with noise that not in the training set. While SEGAN did not remove the novel noise, DAP can better remove noises at the price of lower speech quality in some segments. Please listen to the sounds to appreciate the difference.

---

Under review as a conference paper at ICLR 2020
(a) Input Mixture
(b) Initial Output S,/M;
(c) Intuitive User Input (d) Refined Output S,/M;
Figure 5: Example of interactive DAP. Given sound mixture, our DAP model predict separated sounds and the corresponding mask activation maps. Users can simply draw boxes to interact with our DAP model tell where deactivate activate the predicted masks for refining predicted sounds. As shown, user deactivate a region in the predicted mask for separated sound, and then we obtain better results with the refined mask. All spectrograms are zoomed for visualization.
input 3-sec texture
synthesized 12-second audio texture using DAP
Figure 6: Audio texture synthesis via latent space augmentation. From a 3-second input audio, we interpolate (in green) and extrapolate (in orange) the latent input noise synthesize seamless 12-second audio.
the regions they think should have sound or should be silence for source k. These selected regions would become input activation masks: Mad or deactivate mask: M respectively. The values in the annotated regions are one. To encode these annotations, we introduce mask activation and deactivation losses refine results from our separation networks:
lact Σ MM M and ldeact Σ K
Mr --Mdeaet)
(10)
With the activation and deactivation losses, our DAP framework refines the source generator and masks. This refinement process typically takes of seconds. Figure 5 shows interactive mask editing result. We can see that with adding a mask deactivation loss for the "dog" sound into optimization, our network will remove the "violin" patterns from it.
3.3 AUDIO TEXTURE SYNTHESIS
Another interesting application is audio synthesis audio interpolationnextrapooaaioo The goal of audio synthesis is to lengthen a given audio. This has wide applications in audio texture generation for arbitrary length to match visual content. Here we show example on how we can leverage the input noise latent space to achieve audio texture synthesis (Figure 6).
Given audio texture, we apply DAP framework and obtain the temporally coherent input noises [z', 22 ?...... 2"] for the whole sequence and generator S. To lengthen the input texture, we can first use straightforward interpolation: we can insert more noise frame between every two consecutive
n-5 2² Those symbols in boxes interpolated ones this way we can easily double (or more) the length the input audio. we want to explore diversity in learned generator, can also extrapolate in the latent space.
noise frames: Zint [2 1.5 2 2.5
new

---

Under review as a conference paper at ICLR 2020
Music A Watermark
Music B + Watermark Music C Watermark
Music A
Music
Music
Watermark
Figure 7: DAP can also be applied to automatically remove audio watermark. Top row shows three input mixture audios A, B. and C, all mixed with watermark. Given these 3 mixtures, DAP can seperate each individual sound source out. Note that DAP is only trained on these input mixtures.
In other words, we can create a brand new latent vector that are slightly outside our input training manifold: Zezt = , n++2 ..... 2+m ). These new input latent vectors Zint and Zeat can be used as input to source generator S. Figure 6 shows that we can prolong a 3-second audio texture to 12-second one with a combination of interpolation and extrapolation in latent space. No direct copy and paste is used, so naturally avoid the seam discontinuity problem.
3.4 CO-SEPARATION / AUDIO WATERMARK REMOVAL
Audio watermarks are commonly used in the music industry for seyyright-protccte audios. Supervised deep networks will require lot of training data learn both diverse clean audio patterns and watermark patterns for separating watermark sounds from clean sounds (Muth et al., 2018). Our DAP model can also easily generalize handle co- ro-separation for removing audio watermarks.
Given K sounds:
S2 = S + Swatermark (A: = 1, K) containing the mixture of the same unknown audio watermark Swasemark and the clean audios: Si Sj ..., SK. The goal is to recover the clean audios. Using the proposed separation framework, we learn K + generator networks; K for the music signals and for the watermark with shared network weights and input noise. As shown in Figure 7, from the K input mixtures, can extract the clean music as well as the embedded watermark.
ABLATION STUDY AND DISCUSSION
Temporal Noise Input To capture the temporal coherence prior within audio spectrograms, we decompose individual noise inputs into temporally consistent segments. To justify the temporal noise, we compare to a variant of our model without temporal noise (w/o TN), which has no temporal consistency but with only random per-frame noise. As shown in Figure (b), the model without temporal noise fails to preserve temporal coherent "basketball" sound structures while DAP model can well capture the temporal consistent patterns in the "baskethall" sound.
that
Dynamic Noise We introduce dynamic noise from curriculum learning to capture large variations in individual sounds and preserve temporal consistent structures. In particular, we use abruptly changing dynamic noise during noise input transition for better quickly learning dynamic patterns for individual sound predictions. To show the effects of dynamic noise and abrupt noise transition, we compare to two baseline models, which are without dynamic noise (w/o DN) and without abruptly changing noise (w/o AC), respectively. The results are illustrated in Figure 8 (c) and (d). We see that the w/o DN model fails to separate the two sounds and it can only capture few dynamic structures in the "violin" sound due to its weak dynamic modeling capacity. Although the w/o AC model restore dynamic "violin" patterns, it incorrectly adds "violin" sounds into the "basketball" sounds. In comparison, our full DAP model has strong dynamic modeling capacity for the two individual sound prediction networks immediately.
8

---

Under review as a conference paper at ICLR 2020
baskettal
violin
(a) Input
(b) w/o TN
(c) w/o DN
(d) w/o AC
(e) DAP
(f) GT
Figure 8: Ablation study on noise design. From left to right, spectrograms are from (a) input mixture (b) without temporal noise (w/o TN) (c) without dynamic noise (w/o DN) (d) without abrupt change in dynamic noise (w/o AC) (e) our full DAP model (f) ground truth.
basketball
violin
(a) w/o ID
masks
(b) w/o NZ.
masks
(c) DAP
masks
Figure 9: Ablation study on mask design. From left to right, spectrograms and masks are from models without 1D mask (w/o ID), without nonzero mask loss (w/o NZ), and with both 1D mask and the loss (DAP).
1D Mask Design Unlike commonly used unconstrained masks used in images/videos, design a ID mask. This design explicitly decomposes sound estimation into two sub-problems: temporal sound prediction and mask modulation, which will case mask learning and temporal consistency modeling. Such disentanglement idea has been widely used in separating geometry and appearance estimation (Janner al., 2017 Lin al.) 2019) To validate the effectiveness of the ID mask, we compare it to a baseline model with unconstrained masks. As illustrated in Figure 9 (a), we can see that the model without the strong ID mask constraint easily find short-cut to minimize the loss functions and the unconstrained masks also capture sound content. With the ID mask, our DAP model disentangles sound prediction and mask modulation and successfully separates the two sounds.
Non-Zero Mask To demonstrate the effectiveness of the proposed nonzero mask loss, empirically show results of our DAP without the loss during optimization in Figure 9 (b). From the mask results, we can find that mask values even some sounding regions are close to zero for both two sounds. With the loss term, DAP can reconstruct sounds for all sounding regions.
Discussion RPCA decomposes spectrogram matrix S into low-rank matrix S and sparse matrix S. and S S₁ + S.. To satisfy the principle of the decomposition, St will capture repeating structures in the audio spectrogram and the remaining large variations in the spectrogram will be preserved in the sparse matrix. KAM assumes an audio source at timestep can be estimated using its values at other nearby times through source-specific proximity kernel, allowing addressing local redundancy in audio sources but missing modeling dynamic patterns.
In essence, various formulations have been proposed to utilize temporal redundancy and dynamics. However, it is challenging for a traditional formulation to capture the temporal consistency and dynamic patterns due to its limited capacity. To model both, we take advantage of the large capacity from deep networks and utilize temporal consistent noise inputs, which helps our network simultaneously restore temporal consistent sounds and capture large variations.

---

Under review as a conference paper at ICLR 2020
5 RELATED WORK
As discussed in the introduction, our work is inspired by the recent advances in deep image prior (DIP) and double-DIP (Ulyanov al.) 2018 Gandelsman al., 2019). Recently, MMchelashvilii & Wolf (2019) tried to learn a deep network prior for audio by following DIP process. Instead of learning the priors from audio signals, predicted apriori SNR of input audio is fed into traditional denoising methods, such as LSA (Ephraim & Malah, 1985) or the Weiner filter, to perform audio denoising. Therefore their method still relied on the accuracy of existing denoising algorithms. In our deep audio prior framework, we propose to capture the inherent audio priors using deep networks without relying on or bottlenecked by the previous method.
Audio source separation is classical problem in signal processing (Haykin & Chen 2005 Naik
jet al., 2014, Makino 2018). To address the problem, many blind audio source separation methods have been proposed, such NMF (Virtanen 2007: Smaragdis & Brown 2003). RPCA (Huang et al. 2012). and KAM (Liutkus et al. [2144] Yela et al. 2018). For multichannel audios, independent
component analysis (ICA) is also commonly used (Hyvärinen & Oja, 2000 Smaragdis 1998 Dinh
et al. performance, supervised NMFs are explored in (Mysore & Smaragdis) 2011; Smaragdis 2006).
al. 2014); in our paper, we focus on separation from single-channel audios. To improve separation
However, these methods usually have limited capacity to handle various and complicate sound patterns. In our work, we leverage the large capacity from deep neural networks to encode both the temporal audio priors and large variances that exist in real-world complex audios.
Recently, deep audio separation networks are proposed (Chandna et al. 2017; Hershey et al. 2016
Isik et al. 2016 Chen et 2017 Smaragdis & Venkataramani 2017 Le Roux al.] 2015] Wang & Chen 2018), all of which require a large amount audio training data. Also, a deep model trained certain type of sound sources cannot generalize well audios from unseen categories, shown in $..1. These limitations restrict the existing deep networks to address scalable universal audio separation. Unlike the existing approaches, our DAP framework is only trained the single input mixture audio, does not require any additional training data, and hence is immune from the data mismatch between training and testing sets.
6 CONCLUSION AND FUTURE WORK
We have introduced Deep Audio Prior, a new audio prior framework that requires zero training data. Thanks to the universal and unsupervised nature, are excited about the potential applications that DAP can enable. We have demonstrated impressive results on challenging tasks, even when comparing our model models trained on a large amount of supervised data. All of our examples are listed on the anonymous webpage: https://iclr-dar .ep..ggtti github. 1 Deep-Audio-Prior/
Limitations and Future Work Our framework naturally extends to multiple sound sources, e.g., co-segmentation with 4 sources in total. Yet for audio recorded in the wild, one main challenge is to decide how many effective sound sources are present (Girin et al. 2018). One direction that would like to explore is to the output/error metrics from DAP separation to iteratively decide the optimal number of effective sources. Visual information can also be used to guide this process (Sodoyer al.) 2002), Aytar et al., (016). It remains an open challenge how to design deep audiovisual prior to combine audio and visual information to distill a more robust representation.
In the interactive editing demo ($3.2), we show that the progressive refinement based on user input can be easily done within seconds. However, the initial DAP separation usually takes the order of minutes for a short audio segment. Possible acceleration in the initial training stage is promising direction. In supervised deep learning, distilling knowledge from multiple models has shown great success (Hinton et al. 20155.. Given that DAP works for audios in the wild, we interested in how to robustly distill useful information from a large amount of learned generators from many single audios.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnetr (eds.), Advances in Neural Information Processing Systems 29, pp. 892-900. Curran Associates, Inc., 2016.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, ACM, 2009.
Nicholas J Bryan, Gautham J Mysore, and Ge Wang. Isse: An interactive source separation editor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 257-266. ACM, 2014.
Antonin Chambolle. An algorithm for total variation minimization and applications. Journal of Mathematical imaging and vision, 20(1-2):89 97,9999777777 2004.
Pritish Chandna, Marius Miron, Jordi Janer, and Emilia Gómez. Monoaural audio source separation using deep convolutional neural networks. In International conference on latent variable analysis and signal separation, pp. 258-266 Springer, 2017.
Zhuo Chen, Yi Luo, and Nima Mesgarani. Deep attractor network for single-microphnne speaker separation. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 246-250. IEEE, 2017.
Leon Cohen. Time-frequency analysis, volume 778. Prentice hall, 1995.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 1rXiv:1410.8516 2014.
Yariv Ephraim and David Malah. Speech enhancement using a minimum mean-square error log-spectral amplitude estimator. IEÉE transactions on acoustics, speech, and signal processing, 33(2):443 445, 1985.
Eduardo Fonseca, Jordi Pons Puig, Xavier Favory, Frederic Font Corbera, Dmitry Bogdanov, Andres Ferraro. Sergio Oramas, Alastair Porter, and Xavier Serra. Freesound datasets: a platform for the creation of open audio datasets. In Hu X, Cunningham SJ. Turnbull D, Duan Z, editors. Proceedings of the 18th ISMIR Conference; 2017 7 oct 23-27; Suzhou, China. China./Canatae (Canada]: International Society for Music Information Retrieval; 2017. p. 486-93. International Society for Music Information Retrieval (ISMIR), 2017.
Yossi Gandelsman, Assaf Shocher, and Michal Irani. "double-dip": Unsupervised image decomposition via coupled deep-image-priors In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2019.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780. IEEE, 2017.
Laurent Girin, Sharon Gannot, and Xiaofei Li. Audio source separation into the wild. In Multimodal Behavior Analysis the Wild, Computer Vision and Pattern Recognition, pp. 53-78. Academic Press (Elsevier). November 2018. doi: 10. 10.101///77--- 11----0--11----0 1---1-101---900000000 9.00022-5.
Simon Haykin and Zhe Chen. The cocktail party problem. Neural computation, 177))187551902, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1034-1034, 2015.

---

Under review as a conference paper at ICLR 2020
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31-35. IEEE, 2016.
Shawn Hershey, Sourish Chaudhuri, Daniel W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. Cnn architectures for large-scale audio classification. International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Po-Sen Huang, Scott Deeann Chen, Paris Smaragdis, and Mark Hasegawa-Johnson. Singing-voier separation from monaural recordings using robust principal component analysis. In 2012 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 57-60. IEEE, 2012.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications.
Neurul networks, 13(4-5):411-33) 2000.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 102000100.. 2015.
Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, and John R Hershey. Single-channel multi-speaker separation using deep clustering. arXiv preprint 1rXiv:160..02173. 2016.
Michael Janner, Jiajun Wu, Tejas D Kulkarni, llker Yildirim, and Josh Tenenbaum. Self-supervised intrinsic image decomposition. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 9936-5946 2017.
Aren Jansen, Manoj Plakal, Ratheet Pandya, Daniel PW Ellis, Shawn Hershey, Jiayang Liu, R Channing Moore, and Rif A Saurous. Unsupervised learning of semantic audio representations. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 126-130. IEEE, 2018.
Jonathan Le Roux, John R Hershey, and Felix Weninger. Deep nmf for speech separation. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 66-70. IEEE, 2015.
Chen-Hsuan Lin, Oliver Wang, Bryan C Russell, Eli Shechtman, Vladimir G Kim, Matthew Fisher, and Simon Lucey. Photometric mesh optimization for video bideo-aligned 3d object reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 969-978, 2019.
Antoine Liutkus, Derry Fitzgerald, Zafar Rafi, Bryan Pardo, and Laurent Daudet. Kernel additive models for source separation. IEEE Transactions on Signal Processing, 62(16))429--4310, 2014.
Shoji Makino. Audio Source Separation. Springer, 2018.
Ethan Manilow, Prem Seetharaman, Fatemeh Pishdadian, and Bryan Pardo. NUSSL: the northwestern university source separation library. https://githuu com/
Iiteraattieeucciolaa//mmsss 2018.
Michael Michelashvili and Lior Wolf. Audio denoising with deep network priors. arXiv preprint
arXiv:1904.07612,, 2019.
Pedro Morgado, Nuno Vasconcelos. Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. In Neural Information Processing Systems (NIPS), 2018.
Joachim Muth, Stefan Uhlich, Nathanael Perraudin, Thomas Kemp, Fabien Cardinaux, and Yuki Mitsufuji. Improving dan-based music source separation using phase features. arXiv preprint arXiv: 1807.0807000000 2018.
12

---

Under review as a conference paper at ICLR 2020
Gautham Mysore and Paris Smaragdis. A non-negative approach to semi semi-supervised separation of speech from noise with the use of temporal dynamics. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 17-20. IEEE, 2011.
Ganesh R Naik, Wenwu Wang, al. Blind source separation. Berlin: Springer, 10:978-3, 2014.
Santiago Pascual, Antonio Bonafonte, and Joan Serrà. Segan: Speech enhancement generative adversarial network. Proc. Interspeech 2017, pp. 3642-3646, 2017.
Karol Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd
ACM international conference on Multimedia, pp. 1015-1018. ACM, 2015.
Stuart Rosen. Temporal information in speech: acoustic, auditory and linguistic aspects. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 336 (1278):367-3733 1992.
Paris Smaragdis. Blind separation of convolved mixtures in the frequency domain. Neurocomputingg, 22(1-3):21-34, 1998.
Paris Smaragdis. Convolutive speech bases and their application to supervised speech separation. IEEE Transactions Audio, Speech, and Language Processing, 15(1):1-12, 2006.
Paris Smaragdis and Judith C Brown. Non-negative matrix factorization for polyphonic music transcription. In 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No. 03TH18684, pp. 177-180. IEEE, 2003.
Paris Smaragdis and Shrikant Venkataramani. A neural network alternative to non-negative audio models. In 20/7 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 86-90, IEBE, 2017.
David Sodoyer, Jean-Lue Schwartz, Laurent Girin, Jacob Klinkisch, and Christian Jutten. Separation of audio-visual speech sources: approach exploiting the audio-visual coherence of speech stimuli. EURASIP journal on applied signal processing, 2002 11:1111166-1133 2002.
Martin Spiertz and Volker Gnann. Source-filter based clustering for monaural blind source
separation. In Proceedings of the 12th International Conference on Digital Audio Effects, 2009.
Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. arXiv preprint arXiv: 1rXiv:1806.03185 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Vietor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446-9454, 2018.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE transactions on audio, speech, and language processing, 14(4): 1462-1469, 2006.
Emmanuel Vincent, Rémi Gribonval, and Mark D Plumbley. Oracle estimattrs for the benchmarking of source separation algorithms. Signal Processing, 87(3):1933-199 2007.
Tuomas Virtanen. Monaural sound source separation by nonnegative matrix factorization with temporal continuity and sparseness criteria. IEEE transactions audio, speech, and language processing, 1566):106611074 2007.
DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. IEEÉ/ACM Transactions on Audio, Speech, and Language Processing, 26(10): 2018.
Delia Fano Yela, Dan Stowell, and Mark Sandler. Does k matter? k-nn hubness analysis for kernel additive modelling vocal separation. In International Conference on Latent Variable Analysis and Signal Separation, pp. 280-289. Springer, 2018
Xuaner Zhang, Ren Ng. and Qifeng Chen. Single image reflection separation with perceptual losses. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4786-4794, 2018
13

---

for the UNet can Under review as a conference paper at ICLR 2020
nulf]
Input noise
D1
D2
Dy
Rg(E]
S2
SN
Conv
BN
LeakyReLU
Output
U,
U2
UN
Upsample
Kg[(]
Downsample
Figure 10: Our audio generator and mask generator networks share the same U-Net structure (Ulyanov al. 2018). There are mainly three modules: downsampling module (see D upsampling module (see U,), and skip connection module (see St). Batch normalization (BN) (loffe & Szegedy 2015) used accelerating and stabilizing training and LeakyReLU (He al.) 2015) used as the nonlinearity function. nx[i], n. [t]. n. denote the number of filters at depth 2 for the downsampling. upsampling, and skip-connections respectively. The values kali). [1). k correspond to the respective kernel sizes for the convolutional layers in different modules.
Table Better sample ratio (%) comparing with NME/RPCA/KAM in terms of numerical metries: SDR/SIR/L on Universal-150. All results are larger than 50%, which shows that our DAP can achieve better results than the compared methods on majority of testing samples.
DAP NMF
DAP
RPCA
DAP
KAM
SDR
75.3
66.0
56.0
SIR
76.7
62.7
66.7
LSD
71.4
58.0
71.4
A APPENDIX
A.1 NETWORK STRUCTURE
We use an UNet architecture as in Ulyanov et al. (2018) for our audio and mask generator networks. In the networks, the downsampling is achieved setting the stride of the convolutional layer 2, and the upscaling is implemented by Bilinear interpolation. our implementation, we use relative smaller UNet structure, which only consists of N 3 downsampling modules and upsampling modules. The parameters of the used UNet are n. n. = [16, 32, 64]. Ka = ku
[5, 5, 5]. ns [0, 0, 4), ks =[Nnee, None, 1]. During tuning parameters of the UNet, we have the same observation as Ulyanov et al. (2018) that wide of range of hyper-paramest
achieve similar performance. We empirically found that our models can be converged in 5000 iterations.
A.2 ADDITIONAL SOUND SEPARATION RESULTS
To further clarify the numerical results in the Table 1, except from the mean SDR, mean SIR, and mean LSD, which sensitive over large or over small values (variance also has the same issue), we count the better sample ratio based on the three metrics. The ratio will count our DAP is better than the compared the method on how many testing samples. For example, compared the NMF in terms of SDR, the ratio is 0.753, which shows that our method is better than the NMF on 75.3% testing samples. The comparison results are shown in Table We can see that our DAP achieve better results majority of testing samples comparing to NME/RPCA/KAM.
To further validate our DAP on sound source separation, we compare with other methods on a standard sound separation benchmark from (Vincent et al. (007). which has 20 testing samples and each sample has clean sounds. To evaluate different sound separation methods, we take the first two sounds from each sample to compose 2-sound mixtures. As shown in Table 3, We that
14

---

Under review as a conference paper at ICLR 2020
Table 3: Comparison between DAP/NME/PCCAKAN in terms of numerical metries: SDR/SIR/LSD on standard source separation benchmark (Vincent et al. 2007). For SDR/SIR, higher is better and for LSD, smaller is better. The best results on the three metrics are from our method, DAP.
NMF
RPCA
KAM
Proposed DAP
Spiertz & Gnann 2009
(Huang et al. 2012
Yela ct al.
2018
SDR
-3.36
-4.49
-2.16
-1.37
SIR
-0.93
-1.00
-0.32
0.82
LSD
0.56
0.57
2.19
0.48
Table Better sample ratio (%) comparing with NME/RPCA/KAN in terms of numerical metries:
SDR/SIR/LSD on the source separation benchmark (Vincent et al. 2007). Most ratio results are larger than 50%, which shows that our DAP can achieve overall better results than the compared methods.
DAP vs. NMF
DAP vs. RPCA
DAP vs. KAM
SDR
60.0
70.0
SIR
55.0
40.0
LSD
70.0
65.0
95.0
our method outperforms the three compared methods in terms of SDR, SIR, and LSD. The Table illustrates better ratio results. We can also find that DAP still achieves overall better results.
A.3 DYNAMIC RESULTS
To model dynamic patterns with large variations in audio sources, we we gradually add dynamic noise into inputs as we progress more training iterations. our implementation, the two iteration parameters: Ti and T2 in the Equation (6) are set as 2000 and 4000, respectively. Note that the maximum training iteration number is 5000. When iteration number is smaller than Ti, only temporal coherent noise inputs are used; when iteration number is in [2000, 4000], gradually add dynamic noise into inputs; when iteration number large than 4000, noise inputs are fixed. To illustrate the training dynamics, Figure 11. We can see that our model first restore temporal coherent audio sources with the input noise constraint; when introduce dynamic noise into input (after 2000 iterations), the model will quickly learn the dynamic patterns in the sound mixtures and well capture sound variations while preserving temporal consistent audio sources.
show separation results at different training iterations in
15

---

Under review as a conference paper at ICLR 2020
Modulatte 50u
Geserated sources 500
500
Modulated sounds or 1000 Genernted soarces 1000)
Miskc 1000
Modulated
Modulated
Modulated soundede 2500 Ginerated sourees 2500
Masisse 2500
Modulated soundie 3000
Genernted snarees 30IK
Miskks 3000
Modulated sounds- 3500
Generated sources 3500
Masks 3500
Modulated sounds @ 4000
Genernted sources 4000
Masks
Modulated sound=04500 Generated 6000c
Musksti 45000
Modulated 6 5000 Genernted
Masks 5000
Dynamic results of DAP from different training iterations. We show generated audio sources, masks, and the separated sounds with corresponding mask modulations at 500rh. 1000th, 5000th iterations. respectively. After adding dynamic noise into inputs 2000th iteration, the model will quickly capture the violin sounds with large variations (see results at 2500th).
Figure 11:
16


Reviews:
[{"review_id": "rkl7DDOkcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors detail a set of priors for unsupervised decomposition of individual spectrograms into their component parts. The introduce reasonable constraints on temporal coherence (consistency and dynamic shifts) and mask activations (at least one component always activated). They also regularize sources to not overlap spectrotemporally. Decomposition is performed by training the weights of a U-Net on a single spectrogram as in deep image priors. The authors demonstrate quantitative improvements on blind source separation over other data-agnostic techniques, and qualitative use of the model for interactive editing, audio texture synthesis, and audio watermark removal. The work also performs an ablation study to qualitatively demonstrate the importance of each element for the prior. The experiments are performed well and explained clearly. They also introduce a dataset of diverse mixtures for future comparisons.\n\nPros:\n* Important motivation for why audio has different properties than images (even if it can be represented as a \"image\" spectrogram). The priors are well-motivated by the dynamics of audio.\n* Good ablations and quantitative comparisons to baselines.\n\nCons:\n* Some details could be better demonstrated / explained (even if only in the appendix). For example the paper cites the network architecture, but a local description would be helpful. Similarly, the latent dynamics are carefully regularized, so visualizing them would be helpful to understand the dynamics.\n* The scaling of the technique is not supported by the current experiments. The authors claim they have extended to 4 sources, but all experiments in the paper seem to only involve two sources. \n* More motivation could help in terms of the value of non-amortized methods like deep priors, vs. other approaches such as pretraining or self-supervised methods. While it is difficult to get lots of labeled data for a specific task, the argument was not convincingly made that methods like deep priors should outperform methods that use pretrained priors on adjacent tasks (where collecting data is easy).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Syeteh-f5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces deep audio prior (DAP), which uses CNN'S prior to perform classical tasks in audio processing: source separation, denoising, texture synthesis, co-separation. This paper gets inspiration from deep image prior and adapts it to audio, by introducing a lot of insights in the audio domain, which I believe is a good amount of contribution.\nI have the following questions on the paper:\n(1) The source generation part is a bit confusing. When testing on real-world examples, do you need to generate sources? I want more explanations on the ablations studies in Section 4, temporal/dynamic sources. What are the input/output, how is the noise generated, how is the model trained.\n(2) In general, I expect more details in the paper, like the model architecture (does that affect performance), how training is performed (iteration, convergence, etc).\n(3) Some notations are missing, e.g. equation (4).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lo-8cotH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for blind source separation relying on randomly initialized networks to decompose an input audio spectrogram into two components.\nThe networks are designed to promote temporal contiguity of spectral energy in the estimated signals, which are modulated (in time) by estimated masks.\nThe proposed method is evaluated on a collection of 150 random mixes of sounds, and performs favorably relative to some standard baseline methods (RPCA, NMF, KAM).\n\nThis seems like a promising line of work, but at this point I think the weaknesses of the paper outweigh its strengths (as detailed below).  Some of these points may be addressed during discussion, but I currently lean toward reject.\n\n\nStrengths of the paper:\n\n- The ideas are interesting, and appear to perform well on a simulated and real(ish) data.\n\n- The authors investigate several variations and applications of source separation, including interactive editing, co-separation, and texture synthesis.\n\n- A small (qualitative) ablation study is included to clarify the importance of different components of the loss function.\n\n\nWeaknesses of the paper:\n\n- Much of the presentation is vague or opaque.  There is little detail provided about the specific architectural parameters of the model, and the diagram (figure 1) does not appear to match the equations.  Specifically, it's unclear whether S_1^* is a function of S_mixture or not.\n\n- The quantitative evaluation focuses entirely on one (new) dataset with unclear characteristics.  No details are provided about the evaluation protocol, and in particular, the tuning of hyperparameters for the various methods under comparison.  Aggregate statistics are included (mean? SDR, etc), but no notion of variance or error bars are included.  It's not ultimately clear how fair this evaluation is.  There should at minimum be a comparison on a standard source separation or speech enhancement dataset, in addition to the new set presented here.\n\n- Most of the spectrogram figures appear to be upside-down, which is confusing.  The details of the audio processing are omitted: STFT parameters are stated, but not the sampling rate.\n\n- There are numerous typos (\"grounth\", \"spectrogram stokes\", etc), indicative of the authors not running a spell-checker on their submission.\n\n\nQuestions for the authors:\n\n- The model itself consists of several competing loss functions, but it seems like they may have a trivial, optimal solution at S1 = S_mix (M_1 = 1) and S_2 = 0 (M_2 = 0 or 1).  As far as I can tell, this solution would trivially minimize each term of equation 9.  (If S_1* does not depend on S_mixture, this may be less of an issue, but the trivial solution may still exist when the driving noise is of sufficiently high dimension and the optimization is run long enough.)  Am I misunderstanding the algorithm, or is there some deeper reason why this solution would not be preferred?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use CNN'S prior to deal with the tasks in audio processing. The motivation is weak and the presentation is not clear.  The technical contribution is trivial.
Decision: Reject