Music Source Separation in the Waveform Domain
Authors: Alexandre Defossez
Nicolas Usunier
Leon Bottou
Francis Bach
Venue:  (2020)
Keywords: source separation
audio synthesis
deep learning

Abstract: Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song.Such components include voice, bass, drums and any other accompaniments. While end-to-end models that directly generate the waveform are state-of-the-art in many audio synthesis problems, the best multi-instrument source separation models generate masks on the magnitude spectrum and achieve performances far above current end-to-end, waveform-to-waveform models. We present an in-depth analysis of a new architecture, which we will refer to as Demucs, based on a (transposed) convolutional autoencoder, with a bidirectional LSTM at the bottleneck layer and skip-connections as in U-Networks (Ronneberger et al., 2015). Compared to the state-of-the-art waveform-to-waveform model, Wave-U-Net (Stoller et al., 2018), the main features of our approach in addition of the bi-LSTM are the use of trans-posed convolution layers instead of upsampling-convolution blocks, the use of gated linear units, exponentially growing the number of channels with depth and a new careful initialization of the weights.  Results on the MusDB dataset show that our architecture achieves a signal-to-distortion ratio (SDR) nearly 2.2 points higher than the best waveform-to-waveform competitor (from 3.2 to 5.4 SDR). This makes our model match the state-of-the-art performances on this dataset, bridging the performance gap between models that operate on the spectrogram and end-to-end approaches.

Under review as a conference paper at ICLR 2020
MUSIC SOURCE SEPARATION IN THE WAVEFORM DOMAIN
Anonymous authors Paper under double-bling review
ABSTRACT
Source separation for music is the task of isolating contributions, stems, from dif- ferent instruments recorded individually and arranged together to form song, Such components include voice, bass, drums and any other accompaniments. Contrarily to many audio synthesis tasks where the hest performances are achieved by models that directly generate the waveform, the state-of-the-ar in source separation for mu- sic is to compute masks on the magnitude spectrum. In this paper, we first show that adaptation of Conv-Tasnet (Luo & Mesgarani, 2019), waveform-to-waye model for source separation for speech, significantly beats the state-of-the on the MusDB dataset, the standard benchmark of multi-instrument source separation. Second, we observe that Conv-Tasnet follows a masking approach on the input signal, which has the potential drawback of removing parts of the relevant source without the capacity to reconstruct it. We propose Demucs, a new waveform-to- waveform model, which has architecture closer models for audio generation with more capacity on the decoder. Experiments on the MusDB dataset show that Demucs beats previously reported results terms of signal to distortion ratio (SDR), but lower than Conv-Tasnet Human evaluations show that Demucs has significantly higher quality (as assessed by mean opinion score) than Conv-Tasnet, but slightly more contamination from other sources, which explains the difference in SDR. Additional experiments with larger dataset suggest that the gap SDR between Demucs and Cony-Tasne shrinks, showing that our approach is promising.
INTRODUCTION
Cherry first noticed the "cocktail party effect" (Cherry) 1953); how the human brain is able to separate a single conversation out surrounding noise from a room full people chatting. Bregman later tried to understand how the brain was able to analyse a complex auditory signal and segment it into higher level streams. His framework for auditory seene analysis (Bregman 1990) spawned its computational counterpart, trying reproduce model accomplishments of the brains with algorithmic means (Wang & Brown 2006), in particular regarding source separation capabilities.
When producing music, recordings of individual instruments called stems arranged together and mastered into the final song. The goal source separation is to recover those individual stems from the mixed signal. Unlike the cocktail party problem, there is not single source of interest to differentiate from an unrelated background noise, but instead a wide variety of tones and timbres playing in coordinated way. In the SiSec Mus evaluation campaign for music separation (Stöter et al. 2018), those individual stems were grouped into 4 broad categories: (1) drums, (2) bass, (3) other, (4) vocals. Given music track which is mixture of these four sources, also called the mix, the goal is to generate four waveforms that correspond to each of the original sources. We consider here the case of supervised source separation, where the training data contain music tracks (i.e., mixtures), together with the ground truth waveform for each of the sources.
State-of-the-art approaches in music source separation still operate on the spectrograms generated by the short-time Fourier transform (STFT). They produce mask on the magnitude spectrums for each frame and each source, and the output audio generated by running an inverse STFT on the masked spectrograms reusing the input mixture phase (Takahashi & Mitsufuji 2017| Takahashi et al., (2088). Several architectures trained end-to-end to directly synthesize the waveforms have been proposed (Lliís et al., 2018 Jansson et al. 2017), but their performances far below the otate-of-the-art: in

---

Under review as a conference paper at ICLR 2020
471
611
20
204
254
Figure I: Mel-spectrogram for 0.8 seconds extract of the bass source from the track "Stich Up" of the MusDB test. From left to right: ground truth, Conv-Tasnet estimate and Demues estimate, We observe that Conv-Tasnet missed one note entirely.
the last SiSec Mus evaluation campaign (Stöter et al. 2018). the best model that directly predicts waveforms achieves an average signal-to-noise ratio (SDR) over all four sources of 3.2, against 5.3 for the best approach that predicts spectrograms masks (also see Table in Section 6) An upper bound on the performance of all methods relying on masking spectrograms is given by the SDR obtained when using mask computed using the ground truth sources spectrograms, for instance the Ideal Ratio Mask (IRM) or the Ideal Binary Mask (IBM) oracles. For speech source separation, [Luo & Mesgarani (2019) proposed CanveTasnet, model that reuses the masking approach of spectrogram methods but learns the masks jointly with a convolutional front-end, operating directly in the waveform domain for both the inputs and outputs. Conv-Tasnet surpasses both the IRM and IBM oracles.
Our first contribution is to adapt the Conv-Tasnet architecture, originally designed for monophonic speech separation and audio sampled at 8 kHz, to the task of sterephonic music source separation for audio sampled at 44.1 kHz. Our experiments show that Conv- -asnet outperforms all previous methods by large margin, with SDR of 5.7, but still under the SDR of the IRM oracle at 8.2 (Stöter et al. 2018). However, while Conv-Tasnet separates with a high accuracy the different sources, we observed artifacts when listening to the generated audio: constant broadband noise, hollow instruments attacks or even missing parts. They are especially noticeable on the dir ums and bass sources and we give one such example on Figure T Conv-Tasnet uses an over-complete linear representation on which it applies a mask obtained from a deep convolutional network. Because both the encoder and decoder are linear, the masking operation cannot synthesize new sounds. We conjecture that the overlap of multiples instruments sometimes lead to loss of information that is not reversible by masking operation.
To overcome the limitations of Cony-Tasnet, our second contribution is to propose Demucs, new architecture for music source separation. Similarly to Conv-Tasnet, Demucs a deep learning model that directly operates the raw input waveform and generates waveform for each source. Demues inspired by models for music synthesis rather than masking approaches. It is a U-net architecture with convolutional encoder and a decoder based on wide transposed convolutions with large strides inspired by recent work on music synthesis (Défossez et al.) 2018). The other critical features of the approach are bidirectional LSTM between the encoder and the decoder, increasing the number of channels exponentially with depth. gated linear units as activation function (Dauphin et al. 2017) which also allow for masking, and a new initialization scheme.
We present experiments on the MusDB benchmark, which first show that both Cony-Tasnet and De- mucs achieve performances significantly better than the best methods that operate on the spectrogram, with Conv- Tasnet being better than Demucs in terms of SDR. We also perform human evaluations that compare Conv-Tasnet and our Demues, which show that Demues has significantly better perceived quality. The smaller SDR of Demucs is explained by more contamination from other sources. We also conduct an in-depth ablation study of the Demucs architecture to demonstrate the impact of the various design decisions. Finally, we carry out additional experiments by adding 150 songs to the training set. In this experiment, Demucs and TasNet both achieve an SDR 6.3, suggesting that the gap in terms of SDR between the two models diminishes with more data, making the Demucs approach promising. The 6.3 points of SDR also set a new state-of ofte--of-the of-the-art since it improves on the best previous result of 6.0 on the MusDB test set obtained by training with 800 additional songs.
We discuss in more detail the related work in the next Section. We then describe the original Conv- Tasnet model of Luo & Mesgarani] (2018) and its adaptation to music source separation. Our Demucs
2

---

Under review as a conference paper at ICLR 2020
architecture is detailed in Section 4 We present the experimental protocol in Section 5 and the experimental results compared to the state-of-the-an in Section 6 Finally, we describe the results of the human evaluation and the ablation study.
2 RELATED WORK
first category of methods for supervised music source separation work on frequenyy repre- sentations. They predict a power spectrogram for each source and reuse the phase from the input mixture to synthesise individual waveforms. Traditional methods have mostly focused on blind (unsupervised) source separation. Non-negative matrix factorization techniques (Smaragdis et al.) 2014) model the power spectrum as weighted sum of a learnt spectral dictionary, whose elements are grouped into individual sources. Independent component analysis (Hyvärinen al. (2004) relies on independence assumptions and multiple microphones to separate the sources. Learning soft/binary mask over power spectrograms has been done using either HMM-based prediction (Roweis) 2001) or
segmentation techniques (Bach & Jordan 2005).
With the development of deep learning, fully supervised methods have gained momentum. Initial work was performed on speech source separation (Grais et al. (014), followed by works on music using simple fully connected networks over few spectrogram frames (Uhlich et al. 2015), LSTMs (Uhlich
et al. 2017), multi scale convolutional/recurrent networks (Liu & Yang 2018 Takahashi &
Mitsufuji 2017). Nugraha et al. (2016) showed that Wiener filtering is an efficient post post-processing step for spectrogram- based models and it is now used by all top performing models in this category. Those methods have performed the best during the last SiSec 2018 evaluation (Stöter et al. 2018) for source separation on the MusDB (Rafii et al.) 2017) dataset. After the evaluation, reproducible baseline called Open Unmix has been released by Stöter et al.] (2019) and matches the top submissions trained only on MusDB. MMDenseLSTM, a model proposed by Takahashi et al. al.] (2018) and trained on 807 unreleased songs currently holds the absolute record of SDR in the SiSec campaign. Both Demues and Conv-Tasnet obtain significantly higher SDR.
More recently, models operating in the waveform domain have been developed, so far with worse performance than those operating in the spectrogram domain. A convolutional network with a U-Net structure called Wave-U-Net was used first spectrograms (Jansson et al.) (017) and then adapted to the waveform domain (Stoller al. 2018). Wave-U-Net was submitted the SiSec 2018 evaluation campaign with a performance inferior to that of most spectrogram domain models by large margin.
A Wavenet- -inepiee-is,iedd...... although using a regression loss and not auto-regressive, was first used for speech denoising (Rethage et al. 2018) and then adapted to source separation (Lluís al. 2018). Our model significantly outperforms Wave- UJ-et.Given that the Wavenet inspired model performed worse than Wave-U-Net, we did not consider it for comparison.
In the field of monophonic speech source separation, spectrogram masking methods have enjoyed good performance (Kolbæk et al.. 2017 Isik al., 2016). Luo & Mesgarani (2088) developed waveform domain methods using masking over a learnable front-end obtained from a LSTM that reached the same accuracy. Improvements were obtained by |Wang et al. (2018) for spectrogram methods using the unfolding of a few iterations of a phase reconstruction algorithm in the training loss. In the mean time. Luo & Mesgarani (2019) refined their approach, replacing the LSTM with superposition of dilated convolutions, which improved the SDR and definitely surpassed spectrogram based approaches, including oracles that use the ground truth sources such as the ideal ratio mask (IRM) or the ideal binary mask (IBM). We show in this paper that Conv-Tasnet also outperforms all known methods for music source separation. However suffers from significantly more artifacts than the Demucs architecture we introduce in this paper, as measured by mean opinion score.
3 ADAPTING CONV-TASNET FOR MUSIC SOURCE SEPARATION
We describe this section the Conv-Tasnet architecture of Luo & Mesgarani 2018 and give the details of how we adapted the architecture to fit the setting of the MusDB dataset.
Overall framework Each source s is represented by a waveform x, E RCT where is the number of channels (1 for mono, 2 for stereo) and T the number of samples of the waveform. The mixture (i.e., music track) is the sum of all sources ES Ts. We aim at training a model g
3

---

Under review as a conference paper at ICLR 2020
parameterized by 8, such that g(x) source given 2; that minimizes
(gs (z; 0)) where Is (z; is the predicted waveform for
min E Lgg
(1)
for some dataset D and reconstruction error L. The original Conv-Tasnet was trained using loss called scale-invariant source-to-noiss ratio (SI-SNR), similar to the SDR loss described in Section We instead use simple LI loss between the estimated and ground truth sources. We discuss in more details regression losses in the context of our Demucs architecture Section 4.2
The original Conv-Tasnee architecture Cony-Tasnet (Luo & Mesgarani 2018) is composed of learnt front-end that transforms back and forth between the input monophonic mixture waveform sampled at 8 kHz and 128 channels over-complete representation sampled convolution as the encoder and transposed convolution as the decoder, both with kernel size of 16 and stride of 8. The high dimensional representation masked through separation network composed of stacked residual blocks. Each block is composed of a a 1x1 convolution, PReLU (He) al. 2015b) non linearity, a layer wise normalization over all channels jointly (Ba et al. 2016), a depth- wise separable convolution (Chollet) 2017: Howard et al.) 2017) with a kernel size of 3, a stride and
kHz using
of and a dilation of 2 mad N with n the O-based index of the block and N an another PReLU and normalization. The output of each block participates to the final mask estimation through a skip connection, preceded by a Ixl convolution. The original Conv-Tasmet counted 3 x N blocks with N 8. The mask is obtained summing the output of all blocks and then applying ReLU. The output of the encoder is multiplied by the mask and before going through the decoder.
Conv-Tasnet for music source separation We adapted their architecture the task of stereo- phonic music source separation: the original Conv-Tasnel has receptive field of 1.5 seconds for audio sampled at 8 kHz, we take N 10 and increased the kernel size (resp. stride) of the en- coder/decoder from 16 (resp. 8) to 20 (resp. 10), leading to the same receptive field at 44.1 kHz. We observed better results using 4 N blocks instead of 3 x N and 256 channels for the encoder/decoder instead of 128. With those changes, Conv-Tasned obtained state-of-the-arr performance on the MusDB dataset, surpassing all known spectrogram based methods by a large margin as shown Section 6
Separating entire songs The original few seconds at most. When evaluating it on an entire track, we obtained the best performance by first splitting the input track into chunks of 8 seconds each. We believe this is because of the global layer normalization. During training, only small audio extracts are given, so that a quiet part or loud part would be scaled back to an average volume. However, when using entire songs input, it will most likely contain both quiet and loud parts. The normalization will not map both to the same volume, leading to a difference between training and evaluation. We did not observe any side effects when going from one chunk to the next, so we did not look into fancier overlap-add methods.
model was designed for short sentences of
4 THE DEMUCS ARCHITECTURE
The architecture we propose, which we call Demucs, is described in the next few subsections, and the reconstruction loss discussed in Section 4.2 Demucs takes a stereo mixture as input and outputs a stereo estimate for each source (C = 2). It is an encoder/decoder architecture composed of convolutional encoder, a bidirectional LSTM, and a convolutional decoder, with the encoder and decoder linked with skip U-Net connections. Similarly to other work in generation in both image (Karras al. 20188] (017) and sound (Détossez al. (018),, we do not use batch normalization (loffe & ]zegedy] 2015) as our early experiments showed that it was detrimental to the model performance. The overall architecture is depicted in Figure
4.1 CONVOLUTIONAL AUTO-ENCODER
The encoder is composed of L == 6 stacked convolutional blocks numbered from 1 to L. Each block i is composed of a convolution with kernel size K := 8, stride := 4, C,-1 input channels, C; output channels and ReLU activation, followed by convolution with kernel size 1, 2C,
Encoder
4

---

Under review as a conference paper at ICLR 2020
AMIN CMMOOWN JWn ww0000 MM Ma wm- W www.
Decodes; oupput
w
Reiu/Cons Tridi Coue, N 3,5 1))
Co 20n, R 3,8 11)
Decoder (Cin 100, Cous 4
De Cous
Euroties
Decoder, LSTM
De 5200, Cou 1600))
Linear(Ca 6400,
9200)
Decoder,
Enonder: or LSSM
hidden s2ze=3200
hidirectional layers
GLu(Coardd(Coos 20mz R 1.5 1))
Encudergy 1600, Cour 8500)
Reln(CoowtddcCa Concs K 8,5 4))
Encoder inpat
Encodety(C,, 100, C.n. 200)
Eucodee (C... Can 100)
(b) Detailed view of the layers Decoder, on the top and Encoder; on the bottom. Arrows represent con- nections other parts of the model. For convolutions, Can (resp Cout) is the number input channels (resp output), K the kemel size and the stride.
(a) Demucs architecture with the mixture waveform as input and the four sources estimates as output. Arrows represents U-Net connections.
Figure 2: Demues complete architecture on the left, with detailed representation of the encoder and decoder layers on the right.
output channels and gated linear units (GLU) as activation function (Dauphin et al. 2017). Since GLUs halve the number of channels, the final output of block i has C, output channels. A block is described in Figure 2b Convolutions with kernel width increase the depth and expressivity of the model at low computational cost. As we show in our ablation study 6.2 the usage of GLU activations after these convolutions significantly boost performance. The number of channels in the input mixture is Co = C = 2. while we use C₁ == 100 as the number of output channels for the first encoder block. The number of channels is then doubled at each subsequent block, i.e., C, == 2C for L. so the final number of channels is C2 = 3200. We then use bidirectional LSTM with 2 layers and hidden size Cz. The LSTM outputs 2Cz channels per time position. We use linear layer to take that number down to Cr.
Decoder The decoder is mostly the inverse of the encoder. It composed of L blocks numbered in reverse order from L to 1. The i-th blocks starts with a convolution with stride 1 and kernel width 3 to provide context about adjacent time steps, input/output channels C, and ReLU activation. Finally, we use a transposed convolution with kernel width 8 and stride 4, C;-1 outputs and ReLU activation. The S sources are synthesized at the final layer only, after all decoder blocks. The final layer is linear with S Co output channels, one for each source (4 stereo channels in our case), without any additional activation function. Each of these channels directly generate the corresponding waveform.
U-network structure Similarly to Wave-U-Net (Jansson et al. 2017), there are skip connections between the encoder and decoder blocks with the same index, originally proposed in U-networks (Ronneberger et al., 2015). While the main motivation comes from empirical performances, an advantage of the skip connections is to give direct access to the original signal, and in particular allows to directly transfers the phase of the input signal to the output, as discussed in Section 4.2 Unlike Wave-U-Net, we use transposed convolutions rather than linear interpolation followed by convolution with a stride of 1. For the same increase in the receptive field, transposed convolutions require 4 times less operations and memory. This limits the overall number of channels that can be used before running out of memory. As we observed that a large number of channels was key to obtaining good results, we favored the use of transposed convolutions, as explained in Section 6
Motivation: synthesis vs masking The approach we follow uses the U-Network architecture (Ronneberger et al.] 2015 Stoller et al. 2018 Jansson et al.) 2017), and builds transposed convolutions with large number of channels and large strides (4) inspired by the approach to the synthesis of music notes of Défossez al. (2018). The U-Net skip connections and the gating
5

---

Under review as a conference paper at ICLR 2020
performed by GLUs imply that this architecture is expressive enough represent masks learnt representation of the input signal, in a similar fashion to Conv-Tasnes The Demucs approach is then more expressive than Conv-Tasnet, and its main advantages are the multi-scale representations of the input and the non-linear transformations to and from the waveform domain.
4.2 LOss FUNCTION
For the reconstruction loss (1; in equation | we either use the average mean square enror or average absolute error between waveforms: for waveform Is containing 7 samples and corresponding to source 8, a predicted waveform R. and denoting with a subseript the 1-th sample of waveform, we use one of [ or Tz2:
T
T
Li Ts)
Lt( =
(ist x2,1)²
(2)
T
T
In generative models for audio, direct reconstruction losses on waveforms can pose difficulties because they are sensitive to the initial phases of the signals: two signals whose only difference is shift in the initial phase are perceptually the same, but can have arbitrarily high I. or L.2 losses. can be a problem in pure generation tasks because the initial phase of the signal is unknown, and losses on power/magnitude spectrograms are alternative that do not suffer from this lack of specification of the output. Approaches that follow this line either generate spectrograms (e.g.. [ang et al.) (017). use a loss that compares power spectrograms target/generated waveforms (Défossez et al. 2018).
The problem of invariance to shift of phase is not severe in source separation it is in unconditional generation, because the model has access to the original phase of the signal. The pase can easily be recovered from the skip connections in U-net-style architectures for separation, and is directly used as input of the inverse STFT for methods that generate masks on power spectrograms. As such, losses such L1 /.. are totally valid for source separation. Early experiments with an additional term including the loss of Défossez et al. (2018) did not suggest that it boosts performance, so we did not pursue this direction any further. Most our experiments use L1 loss, and the ablation study presented in Section 6.2 suggests that there is no significant difference between L1 and L2.
4.3 WEIGHT RESCALING AT INITIALIZATION
The initialization of deep neural networks is known to have a critical impact on the overall perfor- mances (Glorot & Bengio 2010 He et al. 2015a), up the point that Zhang al, (2019) showed that with a different initialization called fixup, very deep residual networks and transformers can trained without batch normalization. While Fixup is not designed for U-Net-style skip connections, observed that the following different initialisation scheme had great positive impact on performances compared to the standard initialization of He et al. (2015a) used in U-Networks.
Considering the so-called Kaiming initialization (He et al. 2015a) as a baseline, let us look at a single convolution layer for which we denote the weights after the first initialization. We take := std(w) (w/4 /, where a is a reference scale, and replace w by w' w/Va. Since the original weights have element-wise order of magnitude (KCmn)-1/2 where is the kernel width and Cin the number of output channels, it means that our initialization scheme produces weights of order of magnitude (KCm)) - 1/4, together with non-trivial scale. Based search over the values [0.01, 0.05, 0.1], we select = 0.1 for all the regular and transposed convolutions, see Section for more details. We experimentally observed that on a randomly initialized model applied an audio extract, it kept the standard deviation of the features along the layers of the same order of magnitude. Without initial rescaling, the output the last layer has magnitude 20 times smaller than the first.
4.4 RANDOMIZED EQUIVARIANT STABILIZATION
A perfect source separation model is time equivariant, i.e. shifting the input mixture by X samples will shift the output Y by the exact same amount. Thanks to its dilated convolutions with a stride of 1, the mask predictor of Tonv-Tasnet is naturally time equivariant and even if the encoder/decoder is not strictly equivariant, Conv-Tasne still verifies this property experimentally (Luo & Mesgarani 2019). Spectrogram based method will also verify approximately this property. 'Shifting the input

---

Under review as a conference paper at ICLR 2020
by small amount will only reflect in the phase of the spectrogram. As the mask is computed only from the magnitude, and the input mixture phase is reused, the output will naturally be shifted by the same amount. On the other hand, noticed that architecture did not naturally satisfy this property. We propose a simple workaround called randomized equivariant stabilization, where we sample S random shifts of an input mixture x: and average the predictions of model for each, after having applied the opposite shift. This technique does not require changing the training procedure or network architecture. Using S 10, we obtained 0.3 SDR gain, see Section 6.2 for more details, does make evaluation of the model S times slower, however, on V100 GPU, separating minute of audio 44.1 kHz with Demucs takes only 0.8 second. With this technique, separation of minute takes 8 seconds which is still more than 7 times faster than real time.
5 EXPERIMENTAL SETUP
5.1 EVALUATION FRAMEWORK
MusDB and additional data We use the MusDB dataset (Rafii et al. 2017) which is composed of 150 songs with full supervision in stereo and sampled at 44100Hz. For each song, we have the exact waveform of the drums, bass, other and vocals parts, i.e. each of the sources. The actual song, the mixture, is the sum of those four parts. The first 84 songs form the train set, the next 16 songs form the valid set (the exact split is defined in the musdb python package) while the remaining 50 are kept for the tesf set. We collected raw stems for 150 tracks, i.e., individual instrument recordings used in music production software to make song. We manually assigned each instrument to one of the sources in MusDB. We call this extra supervised data the stem set. We also report the performances of Tasnet and Demucs trained using these 150 songs in addition to the 84 from MusDB, to anaylze the effect of adding more training data.
Source separation metrics Mcasurements of the performance of source separation models was developed by Vincent et al. for blind source separation (Vincent et al. (006) and reused for supervised source separation in the SiSec Mus evaluation campaign (Stöter et al.) 2018). Similarly to previous work (Stoller et al.) 2018 Takahashi & Mitsufujii 2017: Takahashi et al. 2018), we focus on the SDR (Signal to Distortion Ratio) which measures the log ratio between the volume of the estimated source projection onto the ground truth, and the volume of what is left out of this projection, typically contamination by other sources or artifacts. Other metries can be defined (SIR and SAR) and we present them in the supplementary material. We used the python package museval which provide a reference implementation for the SiSec Mus 2018 evaluation campaign. As done in the SiSec Mus competition, we report the median over all tracks of the median of the metric over each track computed using the museval package.
5.2 BASELINES
As baselines, we selected Open Unmix (Stöter et al. 2019), 3-layer BiLSTM model with encoding and decoding fully connected layers on spectrogram frames. was released by the organizers of the SiSec 2018 to act strong reproducible baseline and matches the performances of the best candidates trained only on MusDB. We also selected MMDensel (Takahashi et al. 2018). a multi-band dense net with LSTMs different scales of the encoder and decoder. This model was submitted as TAK2 and trained with 804 extra labeled [ Both MMDenseLSTM and Open Unmix use Wiener filtering (Nugraha et al., 2016) as a last post processing step. The only waveform based method submitted to the evaluation campaign is Wave-U-Net (Stoller et al. 2018) with the identifier STL2. Metries were downloaded from the SiSec submission repository. for Wave-U- -etet and MMDenseLSTTM For Open Unmix they were provided by their author: We also provide the metrics for the Ideal Ratio Mask oracle (IRM), which computes the best possible mask using the ground truth sources and is the topline of spectrogram based method (Stöter et al. 2018).
https://aendd.rrr//rer//////6
7

---

Under review as a conference paper at ICLR 2020
Table I: Comparison of Conv-Tasnet and Demucs to state-of-the-art models that operate on the waveform (Wave-U-Net) and on spectrograms (Open-Unmix without extra data, MMDenseLSTT with extra data) as well as the IRM oracle on the MusDB test set. The Extra? indicates the number of extra training songs used, We report the median over all tracks of the median SDR over each track, as done the SiSec Mus evaluation campaign (Stöter et al. 2018). The All column reports average over all sources. Demucs metrics are averaged over 5 runs, the confidence interval is the standard deviation over v5. In bold are the values that are statistically state-of-the-ar either with or without extra training data.
Test SDR in dB
Architecture
Wav?
Extra?
All
Drums
Bass
Other
Vocals
IRM oracle
N/A
8.22
8.45
7.12
7.85
9.43
Open-Unmix
X
X
5,33
5.73
5.23
4.02
6,32
Wave-U-Net
X
3.23
4.22
3.21
2.25
3,25
Demucs
X
5.58
2.03
1:06 6.08
5.83 2.07
4.12 1.04
6,29 1.00
Conv-Tasnes
X
5.73
2.03
6.08
1:00
5.66 1.16
4.37 4.02
6,81 4:04
Demues
150
6.33
0.0
7.08
.000
6.70
0.06
4.47
7.05
1:04
Conv-Tasnet
150
6.32 :.01
7.11 13
7.00 2.05
4.44
6.74 #.06
MMDenseLSTM
804
6.04
6.81
5.40
4.80
7.16
5.3 TRAINING PROCEDURE
Epoch definition and augmentation We define epoch the dataset pass over all I--second extracts with a stride of second. We use a random audio shift between 0 and second and keep 10 seconds of audio from there augmentation (Uhlich et al. 2017), also used by Open Unmix and MMDenseLSTM. shuffling sources within one batch to generate one new mix, randomly swapping channels. We additionally multiply each source by +1 (Nachmani & Wolf 2019). All Demucs models were trained over 240 epochs. was trained for 360 epochs when trained only on MusDB and 240 when trained with extra data and using only 2-seconds audio extracts.
training example. We perform the following data
Training setup and hyper-parameters All models are trained with V100 GPUs with 32GB of RAM. We use a batch size of 64, the Adam (Kingma & Ba 2015) optimizer with learning rate was chosen among [3e-4, 5e-4] and the initial number of channels was chosen in [64, 80, 100) based on the L1 loss the validation set. We obtained best performance with a learning rate of 3e 4 and 100 channels. We then tried 3 different values for the initial weight rescaling reference level described in Section 4.3 [0.01, 0.05, 0.1] and selected 0. We computed confidence intervals using 5 random seeds in Table For the ablation study on Table 4 we provide metrics for single run.
6 EXPERIMENTAL RESULTS
In this section, we first provide experimental results on the MusDB dataset for Conv-Tasee and Demues compared with state-of-the-art baselines. We then dive into the ablation study of Demues.
6.1 COMPARISON WITH BASELINES
We provide comparison the state-of -f----f-thhhhee the-art baselines on Table The models on the top half were trained without any extra data while the lower half used unreleased training songs. no previous work included confidence intervals, we considered the single metric provided by for the baselines as the exact estimate their mean performance.
Quality of the separation We first observe that Demucs and methods for music source separation. Conv-Tasnet has significantly higher SDR with 5.73, improving by 0.4 over Open-Unmix. Our proposed Demues architecture has worse overall performance but
outperforms all previous
8

---

Under review as a conference paper at ICLR 2020
Table 2: Mean Opinion Scores (MOS) evaluating the quality and absence of artifacts of the separated audio. 38 people rated 20 samples each, randomly sample from one of the 3 models or the ground truth. There is one sample per track in the MusDB test set and each seconds long. Ratings of means that the quality is perfect (no artifacts).
Quality Mean Opinion Score
Architecture
A11
Drums
Bass
Other
Vocals
Ground truth
4.46 0.07
4.56 +.13
4.25
#.15
4.45 #.13
4.64 #.13
Open-Unmix
3.03 1.00
3.10 1.17
2.93 1.20
3.09 10.16
3.00 1.17
Demucs
3.22 1.09
3.77 1.15
3.26
1.18
3.32 1.15
2.55 1.20
Conv-Tasnet
2.85 2.08
3.39
#.14
2.29
2.15
3.18
#.14
2.45 2.16
Table 3: Mean Opinion Scores (MOS) evaluating contamination by other sources. 38 people rated 20 samples each, randomly sampled from one of the 3 models or the ground truth. There is one sample per track in the MusDB test and each is 8 seconds long. Ratings of means no contamination by other sources.
Contamination Mean Opinion Score
Architecture
Drums
Bass
Other
Vocals
Ground truth
4.59 +.07
4.44 1.18
4.69 +.09
4,46 1.13
4.81 4:11
Open-Unmin
3.27 #.11
3,02 #.19
4.00 +.20
3.11 #.21
2.91 +:20
Demucs
3.30 +.10
3.08 +.21
3.93 1.18
3.15 #.19
3.02 2:20
Cony-Tasme 3.42 1.09
3.37 1.17
3.73 1.18
3.46 1.17
3.10 1.17
matches Conv- for the drums source and surpasses for the bass. When training on 150 extra songs, the two methods have the same overall performance of 6.3 SDR, beating MMDenseLSTMM by nearly 0.3 SDR, despite MMDenseLSTT being tained on 804 extra songs. Unlike for speech separation (Luo & Mesgarani 2019). all methods still far below the IRM oracle, leaving room for future improvements. We provide results for the other metrics (SIR and SAR) as well as box plots with quantiles over the test set tracks in Appendix B Audio samples for Demucs, Cony-Tasner and all baselines are provided in the ICLR link code, with more details given in Appendix
Human evaluations We noticed strong artifacts on the audio separated by Conv-Tasnet, especially for the drums and bass sources: static noise between and 2 kHz, hollow instrument attacks or missing notes as illustrated on Figure [] In order to confirm this observation, we organized a mean opinion score survey. We separated seconds extracts from all the test set tracks for Conv-Tasnet, Demucs and Open-Unmix. We asked 38 participants to rate 20 samples each, randomly taken from one of the models or the ground truth. For each one, they were required to provide 2 ratings many artifacts and distortion, content is hardly recognizable, 5: serrectt quality, no artifacts) and the second one evaluated contamination by other sources (1: contamination if frequent and loud, 5: contamination). We show the results on Tables and We confirmed that the presence of artifacts in the output of Conv- degrades the user experience, with a rating of 2.85 08 against 3.22 + .09 for Demucs. On the other hand, Conv-Tasnet samples had less contamination by other sources than Open-Unmix or Demues, although by a small margin, with a rating of 3.42 + .09 against 3.30 .10 for Demucs and 3.27 + 11 for Open-Unmix. Unmix.
scale of 1 to 5. The first one evaluated the quality and absence of artifacts (1:
Training speed We measured the time taken to process single batch of size 16 with 10 seconds of audio at 44.1kHz (the original Wave-U-No U-Net being only trained on 22 kHz audio, we double the time for fairness), ignoring data loading and using torch. cuda. synchronize to wait all kernels to be completed. MMDenseLSTT does not provide a reference implementation. Wave-U-Ne takes 1. seconds per batch, Open Unmix 0.2 seconds per batch and Demucs 1.6 seconds per batch.

---

Under review as a conference paper at ICLR 2020
Table 4: Ablation study for the novel elements in our architecture described in Section 4 only the train set from MusDB and report best L1 loss over the valid set throughout training as well the SDR on the test set for the epoch that achieved this loss.
We use
Valid set
Test set SDR
Difference
LI loss
initial weight rescaling
0.172
4.94
no BiLSTM
0.175
5.12
ReLU instead of GLU
0.177
5.19
no 1xl convolutions in encoder no randomized equivariant stabilization
0.176
5.30
N/A
5.34
kernel size of in decoder convolutions
0.166
5.51
MSE loss
N/A
5.55
Reference
0.164
5.58
Conv-Tasnet cannot be trained with such large sample size, however a single iteration over 2 seconds of audio with a batch size of takes 0.7 seconds.
6.2 ABLATION STUDY FOR DEMUCS
We provide an ablation study of the main design decisions for Demues in Table Given the cost of training a single model, we did not compute confidence intervals for each variation. Yet, any difference inferior .06, which the standard deviation observed over 5 repetitions of the Reference model, could be attributed to noise.
We observe small not significant improvement when using the loss instead of the MSE loss. Adding BiLSTM and using the initial weight rescaling described in Section 4.3 provides significant gain, with an extra 0.48 SDR for the first and 0.64 for the second. We observe that using randomized equivariant stabilization described in Section gives a gain of almost 0.3 SDR. We did not report the validation loss as we only use stabilization when computing the SDR over the test set. We applied the randomized stabilization to Open- Open-Unmix and Conv-Tasnel with no gain, since, as explained in Section 4.4 both are naturally equivariant with respect to initial time shifts.
We introduced extra convolutions in the encoder and decoder, as described in Sections two proved useful, improving the expressivity of the model, especially when combined with GLU activation. Using kernel size of instead of in the decoder further improves performance. We conjecture that the context from adjacent time steps helps the output of the transposed convolutions be consistent through time and reduces potential artifacts arising from using a stride of 4.
The
CONCLUSION
We showed that Conv-Tasnet, state-of-the-art architecture for speech source separation that predicts masks on learnt front-end over the waveform domain, achieves state-of sitie-of-theaaae ti-thh-art art performance for music source separation, improving over all previous spectrogram or waveform domain methods by 0.4 SDR. While Conv- -anneTt has excellent performance to separate sources, it suffers from noticeable artifacts as confirmed by human evaluations. We developed an alternative approach, Demues, that combines the ability to mask over learnt representation with stronger decoder capacity that allows for audio synthesis. We conjecture that this can be useful when information lost the mix of instruments and cannot simply be recovered by masking. We show that our approach produces audio of significantly higher quality as measures by mean opinion scores and matches the SDR of Conv-Tasse when trained with 150 extra tracks. We believe those results make it promising alternative to methods based on masking only.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. Technical Report 1607.06450, arXiv, 2016.
Francis Bach and Michael I. Jordan. Blind one-microphone speech separation: A spectral learning approach. In Advances in neural information processing systems, 2005.
A. Bregman. Auditory Scene Analysis. MIT Press, Cambridge, MA, 1990.
E. Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. The Journal of the Acoustic Society of America, 1953.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the International Conference on Machine Learning, 2017.
Alexandre Défossez, Neil Zeghidour, Usunier Nicolas, Leon Bottou, and Francis Bach. Sing: Symsoo-to-instrummet neural generator. In Advances in Neural Information Processing Systems 32, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, 2010.
Emad M. Grais, Mehmet Umut Sen, and Hakan Erdogan. Deep neural networks for single channel source separation. In International Conference on Acoustic, Speech and Signal Processing (ICASSP), 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenct classification. In Proceedings of the IEEE international conference on computer vision, 2015a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, 2015b.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. Technical Report 1704.04861, arXiv, 2017.
Aapo Hyvärinen, Juha Karhunen, and Erkki Oja. Independent component analysis. John Wiley & Sons, 2004.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Technical Report 1502.03167, arXiv, 2015.
Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, and John R Hershey. Single-channel multi-speaker separation using deep clustering. Technical Report 1607.02173, arXiv, 2016.
Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel Bittner, Aparna Kumar, and Tillman Weyde. Singing voice separation with deep u-net convolutional networks. In ISMIR 2018, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. Technical Report 1710.10196, arXiv, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Technical Report 1812.04948, arXiv, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
11

---

Under review as a conference paper at ICLR 2020
Morten Kolbæk, Dong Yu, Zheng-Hua Tan, Jesper Jensen, Morten Kolbaek, Dong Yu, Zheng Hua Tan, and Jesper Jensen. Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks. IEEE/ACM Transactions Audio, Speech and Language Processing (TASLPP) 2017.
Jen-Yu Liu and Yi-Hsuan Yang. Denoising auto-encoder with recurrent skip connections and residual regression for music source separation. 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), 2018.
Francesc Lluís, Jordi Pons, and Xavier Serra. End-to-end music source separation: is it possible in the waveform domain? Technical Report 1810.12187, arXiv, 2018.
Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single- channel speech separation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time feee-feryenccy magnitude masking for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019.
Eliya Nachmani and Lior Wolf. Unsupervised singing voice conversion. Technical Report 1904.06590, arXiv, 2019.
Aditya Arie Nugraha, Antoine Liutkus, and Emmanuel Vincent. Multichannel music separation with deep neural networks. In Signal Processing Conference (EUSIPCO), 2016 24th European. IEEE, 2016.
Zafar Rafii, Antoine Liutkus, Fabian-Rober Stöter, Stylianos loannis Mimilakis, and Rachel Bittner The musdble 18 corpus for music separation, 2017.
Dario Rethage, Jordi Pons, and Xavier Serra. A wavenet for speech denoising. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention. 2015.
Sam T. Roweis. One microphone source separation. In Advances in Neural Information Processing Systems, 2001.
Smaragdis, C. Fevotte, G. J. Mysore, N. Mohammadiha, and M. Hoffman. Static and dynamic source separation using nonnegative factorizations: A unified view. IEEE Signal Processing Magazine, 31()), 2014.
Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. Technical Report 1806.03185. arXiv, 2018.
F.R. Stöter, S. Uhlich, A. Liutkus, and Y. Mitsufuji. Open-unmix a reference implementation for music source separation. Journal of Open Source Software, 2019.
Fabian-Robert Stöter, Antoine Liutkus, and Nobutaka Ito. The 2018 signal separation evaluation campaign. In 14th International Conference Latent Variable Analysis and Signal Separation, 2018.
Naoya Takahashi and Yuki Mitsufuji. Multi-scale multi-band densenets for audio source separation. In Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEFE, 2017.
Naoya Takahashi, Nabarun Goswami, and Yuki Mitsufuji. Mmdenselstm: An efficient combination of convolutional and recurrent neural networks for audio source separation. Technical Report 1805.02410, arXiv, 2018
Stefan Uhlich, Franck Giron, and Yuki Mitsufuji. Deep neural network based instrument extraction from music. In International Conference on Acoustics. Speech and Signal Processing (ICASSP), 2015.
12

---

Under review as a conference paper at ICLR 2020
Stefan Uhlich, Marcello Porcu, Franck Giron, Michael Enenkl, Thomas Kemp, Naoya Takahashi, and Yuki Mitsufuji. Improving music source separation based on deep neural networks through data augmentation and network blending. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech and Language Processing, 2006.
DeLiang Wang and Guy J. Brown (eds.). Computational Auditory Scene Analysis. IEEE Press, Piscataway, NJ, 2006.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. Technical Report 1703.10133 arXiv, 2017.
Zhong- Qiu Wang, Jonathan Le Roux, DeLiang Wang. and John R Hershey. End-to-end speech separation with unfolded iterative phase reconstruction. Technical Report 1804,,0204, arXiv, 2018.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization, arXiv preprint
2019.
13

---

Under review as a conference paper at ICLR 2020
APPENDIX
A
AUDIO SAMPLES
We provide audio samples taken from the test set of MusDB. They are available through the ICLR code sharing url along with all the source code to reproduce our experiments. The audio files for the Wave-U-Net and MMDenseLSTM have been obtained from the SiSec Mus 2018 evaluation campaign results website For Open Unmix, we generated them from the pretrained UMX model using the reference Py PyTcch implementation We recommend listening the audio samples with headphones, while being careful with the volume. An HTML page index. html is provided for easier comparison. The following folders provided:
Reference: ground truth,
Open Unmix,
UNeeUNet,
Demucs: trained only on MusDB,
DemucsExtra: trained on MusDB and an extra 150 songs,
trained only on MusDB,
ContTassetxxtra:: trained on MusDB and an extra 150 songs.
MMDDessNNetLSSTMM trained on MusDB and an extra 804 songs.
B RESULTS FOR ALL METRICS WITH BOX PLOTS
Reusing the notations from Vincent et al. (2006). let us take a source j E 1, 2, 3, 4 and introduce P., (resp Pa) the orthogonal projection sj (resp Span(s1...... We then take with & the estimate of source 8j
Starget
Pas (8))
Cinterf
(a
(8)(
Cartif åj Pa(8g)
We now define various signal noise ratio, expressed in decibels (dB): the source to distortion ratio
Starget 11?
SDR 10 log
|eette + Cartiell"
the source interference ratio
Starget
SIR == 10 log1o
and the sources to artifacts ratio
Starget + Cinterl
SAR = 10
| ||
As explained in the main paper, extra invariants are added when using the museval package. We refer the reader Vincent al.) (2006) for more details. We provide box plots for each metric and each target on Figure 3] generated using the notebook provided specifically by the organizers of the SiSec Mus evaluation campaigr") Hereafter, we provide the equivalent of Table in the main paper for both SIR and SAR.
d1=1
httts://slleec|.ummix...8
https://qithul httea////itbbbcccooo///////////t
14

---

Under review as a conference paper at ICLR 2020
Test SIR in dB
Architecture
Wav?
Extra?
A11
Drums
Bass
Other
Vocals
IRM oracle
N/A
15.53
15.61
12.88
12.84
20.78
Open-Unmin
10.49
11.12
10.93
6.59
13.33
Wave-U-Net
X
6.26
8.83
5.78
2.37
8.06
Demucs
X
10.39 #.07
11.81 2.27
10.55 20
5.90 =.04
13.31 #.21
Conv-Tasnet
X
11.47 +.09
12.31 #.09
11.52 15
7.76 =.07
14.30 #.32
Demucs
150
11.95 4.00
13.74
13.03
7.11 =.10
13.94
Conv-Tasnet
150
12.24 0.00
13.66 1.14
13.18 1.13
8.40 =008
13.70 #.22
MMDenseLSTM
804
12.24
11.94
11.59
8.94
16.48
Test SAR in dB
Architecture
Wav?
Extra?
A11
Drums
Bass
Other
Vocals
IRM oracle
N/A
8.31
8.40
7.40
7.93
9.51
Open-Unmix Unmix
X
5,90
6.02
6.34
4.74
6.52
Wave-U-Net
X
4.49
5.29
4.64
3.99
4.05
Demues
X
2.01 6.08
6.18 1.03
6.41 2.05
5.18 1.06
6,54 1:004
Conv-Tasnet
X
6.13
=.01
6.19 05
6.60 =.07
4.88 + 0.02
6.87 #.05
Demucs
150
6.50
1.02
7.04 1.07
6.68
1.04
5.26
1.03
7.00 1105
Conv-Tasnet
150
6.57 1.02
7.35 1.05
6.96 1.08
4.76 1.05
7.20 1.005
MMDenseLSTM
804
6.50
6.96
6.00
5.55
7.48
15

---

Under review as a conference paper at ICLR 2020
target drums | metric
SDR
target drums
| metric SIR
target drums metric SAR
ConTTasnelExtra
DemucsExtra
SMDenseLSTM
method
Demucs
CanvTasnet
OpenUnmix
WaveUNet
10
15
10
20
0
10
15
score
target bass | metric = SDR
target bass | metric = SIR
target bass | metric = SAR
DemucsExtra
ConTTasnelExtra
ConVTasnet
method
Demucs
MMDenseLSTM
OpenUnmix
WaveUNot
5
10
15
-10
10
20
30
0
5
10
15
score
score
score
target other metric SDR
target other metric SIR
target other metric SAR
MMDenseLSTM
ConTTasnetExtra
method
OpenUnmix
Demucs
WaveUNet
-255
0.0
2.5
5.0
7.5
-5
15
5
score
score
score
target vocals metric SDR
target vocals metric SIR
target vocals metric SAR
MMDenseLSTM
DemucsEExra
ConvTasnet
method
ConTTasnatExtra
OpenUnmix
Demucs
WaveUNNet
10
score
score
score
Figure 3: Boxplot showing the distribution of SDR, SIR and SAR over the tracks of the MusDB test.
16


Reviews:
[{"review_id": "SJxIHvP15H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggests using a Unet type architecture to perform end-to-end source separation. They are reporting performance improvement over another architecture which uses Unets architecture in conjunction with Wavenet decoders. They also report a very marginal performance improvement over an STFT based model (open unmix) The improment is 0.02 dB (table 1), and I am not sure if it is statistically significant.\n\n\nAlso, there is no mention of algorithms which adaptively learn the basis and then do masking similar to what we do in the STFT domain. A very popular example for this is tasnet, which performs well on speech source separation tasks. I would like to see comparisons with this model. If you think there is a specific reason why not to use adaptive basis approaches such as TASNET, please do let me know. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkl_LuUQcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors consider the task of supervised music source separation, i.e., separating out the components (bass, drums, voice, other) out of a mixed music track. In particular, the work considers the task of supervised source separation where the individual target tracks are available during training time. The main contribution of this work is the improvement of an end-to-end waveform-to-waveform separation models through a number of architectural changes that allow such waveform-to-waveform models to perform comparably with other current state-of-the-art methods that instead operate in the spectrogram domain.\n\nOverall, the paper is generally well written and the method is easy to follow. However, I have a couple of concerns about this work, based on which I would rate the work as “weak reject”:\n\n1. The specific architecture proposed in this task does appear to improve performance for this particular task, but it is not clear to me that the conclusions drawn based on the study in this work will be generally applicable to other related tasks. Also, the final model is still only comparable to the spectrogram-based methods overall, and appears to do a slightly worse job in separating the ‘vocal’ and ‘other’ tracks than these baseline methods. As such, I’m unsure about the impact of this work.\n2. The section describing the evaluation metric SDR wasn’t very clear to me. Perhaps it would be better to just refer back to (Vincent et al., 06) and just describe what SDR captures instead? Or alternatively, more details could be added to explain the computation more clearly.\n3. Section 4.2: The authors mention that they multiply each source by +/- 1. I didn’t follow why this is being done. Could the authors please clarify.\n\nMinor comments:\n1. The notation in Equation 1 is slightly confusing because ‘s’ is used to index both the mixture and the sources. I think x \\in \\mathcal{D} would be clearer. Similarly I wonder if x_s \\in \\mathbb{R}^C \\times \\mathbb{R}^T would be clearer than x_s \\in \\mathbb{R}^{C,T}\n2. The reference (Oord et al., 2017) should be rendered as (van den Oord et al., 2017) \n3. The Figure uses K, and S to denote Kernel Width and stride, but this isn’t explicitly mentioned in the text. Perhaps it would be useful, in Section 3.1 to write: “... convolution with kernel width, K=8, and stride, S=4, ...” or something similar, and updating the caption to explain the notation.\n4. Section 3.2: “The decoder is almost the symmetric of the encoder” --> “The decoder is almost the inverse of the encoder” would perhaps be better?\n5. Section 3.2: “The final layer … S.C_0 ...” --> “The final layer … S * C_0 ...”\n6. The notation in Equation 1 and Equation 2 is slightly inconsistent (lower-case vs. upper-case L for the loss function)\n7. Section 4.3: “additionnaly” --> “additionally”; “32 GB or RAM” --> “32 GB of RAM”\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g-lHiCKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present modifications to the state of the art (waveform based) source separation model (Wave-U-Net) and improve the state of the art to be comparable to spectral masking based methods. They clearly outline all the architectural changes they make (GLU nonlinearities, strided upsampling, bidirectional RNN), perform thorough evaluations against strong baselines, and a complete ablation study to demonstrate the value of each component. The paper reads well and quickly informs newcomers to the field with proper motivation and context. For all these reasons I think it should be accepted.  One weakness of the paper is that the relative incremental nature of the study, but given the thoroughness and clarity of the experiments, and the non-triviality of the architecture search, I think this is a valuable contribution for the ICLR community.\n\nOne small suggestion, the language difference between \"upsampled\" convolution and \"transposed\" convolution is a bit confusing. I might suggest focusing on the striding of the convolution vs. bilinear upsampling, as a non-strided convolution can still be \"transposed\" from a standard API point of view in pytorch or tensorflow.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a waveform-to-waveform music source separation system. Experimental justification shows the proposed model achieved the best SDR among all the existing waveform-to-waveform models, and obtained similar performance to spectrogram based ones. The paper is clearly written and the experimental evaluation and ablation study are thorough. But the main concern is the limited novelty, it is an improvement over the existing Wave-U-Net, it added some changes to the existing model architecture for better modeling the waveform data and compared masking vs. synthesis for music source separation.  
Decision: Reject