Music Source Separation in the Waveform Domain

1 INTRODUCTION . Cherry first noticed the “ cocktail party effect ” ( Cherry , 1953 ) : how the human brain is able to separate a single conversation out of a surrounding noise from a room full of people chatting . Bregman later tried to understand how the brain was able to analyse a complex auditory signal and segment it into higher level streams . His framework for auditory scene analysis ( Bregman , 1990 ) spawned its computational counterpart , trying to reproduce or model accomplishments of the brains with algorithmic means ( Wang & Brown , 2006 ) , in particular regarding source separation capabilities . When producing music , recordings of individual instruments called stems are arranged together and mastered into the final song . The goal of source separation is to recover those individual stems from the mixed signal . Unlike the cocktail party problem , there is not a single source of interest to differentiate from an unrelated background noise , but instead a wide variety of tones and timbres playing in a coordinated way . In the SiSec Mus evaluation campaign for music separation ( Stöter et al. , 2018 ) , those individual stems were grouped into 4 broad categories : ( 1 ) drums , ( 2 ) bass , ( 3 ) other , ( 4 ) vocals . Given a music track which is a mixture of these four sources , also called the mix , the goal is to generate four waveforms that correspond to each of the original sources . We consider here the case of supervised source separation , where the training data contain music tracks ( i.e. , mixtures ) , together with the ground truth waveform for each of the sources . State-of-the-art approaches in music source separation still operate on the spectrograms generated by the short-time Fourier transform ( STFT ) . They produce a mask on the magnitude spectrums for each frame and each source , and the output audio is generated by running an inverse STFT on the masked spectrograms reusing the input mixture phase ( Takahashi & Mitsufuji , 2017 ; Takahashi et al. , 2018 ) . Several architectures trained end-to-end to directly synthesize the waveforms have been proposed ( Lluís et al. , 2018 ; Jansson et al. , 2017 ) , but their performances are far below the state-of-the-art : in the last SiSec Mus evaluation campaign ( Stöter et al. , 2018 ) , the best model that directly predicts waveforms achieves an average signal-to-noise ratio ( SDR ) over all four sources of 3.2 , against 5.3 for the best approach that predicts spectrograms masks ( also see Table 1 in Section 6 ) . An upper bound on the performance of all methods relying on masking spectrograms is given by the SDR obtained when using a mask computed using the ground truth sources spectrograms , for instance the Ideal Ratio Mask ( IRM ) or the Ideal Binary Mask ( IBM ) oracles . For speech source separation , Luo & Mesgarani ( 2019 ) proposed Conv-Tasnet , a model that reuses the masking approach of spectrogram methods but learns the masks jointly with a convolutional front-end , operating directly in the waveform domain for both the inputs and outputs . Conv-Tasnet surpasses both the IRM and IBM oracles . Our first contribution is to adapt the Conv-Tasnet architecture , originally designed for monophonic speech separation and audio sampled at 8 kHz , to the task of sterephonic music source separation for audio sampled at 44.1 kHz . Our experiments show that Conv-Tasnet outperforms all previous methods by a large margin , with an SDR of 5.7 , but still under the SDR of the IRM oracle at 8.2 ( Stöter et al. , 2018 ) . However , while Conv-Tasnet separates with a high accuracy the different sources , we observed artifacts when listening to the generated audio : a constant broadband noise , hollow instruments attacks or even missing parts . They are especially noticeable on the drums and bass sources and we give one such example on Figure 1 . Conv-Tasnet uses an over-complete linear representation on which it applies a mask obtained from a deep convolutional network . Because both the encoder and decoder are linear , the masking operation can not synthesize new sounds . We conjecture that the overlap of multiples instruments sometimes lead to a loss of information that is not reversible by a masking operation . To overcome the limitations of Conv-Tasnet , our second contribution is to propose Demucs , a new architecture for music source separation . Similarly to Conv-Tasnet , Demucs is a deep learning model that directly operates on the raw input waveform and generates a waveform for each source . Demucs is inspired by models for music synthesis rather than masking approaches . It is a U-net architecture with a convolutional encoder and a decoder based on wide transposed convolutions with large strides inspired by recent work on music synthesis ( Défossez et al. , 2018 ) . The other critical features of the approach are a bidirectional LSTM between the encoder and the decoder , increasing the number of channels exponentially with depth , gated linear units as activation function ( Dauphin et al. , 2017 ) which also allow for masking , and a new initialization scheme . We present experiments on the MusDB benchmark , which first show that both Conv-Tasnet and Demucs achieve performances significantly better than the best methods that operate on the spectrogram , with Conv-Tasnet being better than Demucs in terms of SDR . We also perform human evaluations that compare Conv-Tasnet and our Demucs , which show that Demucs has significantly better perceived quality . The smaller SDR of Demucs is explained by more contamination from other sources . We also conduct an in-depth ablation study of the Demucs architecture to demonstrate the impact of the various design decisions . Finally , we carry out additional experiments by adding 150 songs to the training set . In this experiment , Demucs and TasNet both achieve an SDR of 6.3 , suggesting that the gap in terms of SDR between the two models diminishes with more data , making the Demucs approach promising . The 6.3 points of SDR also set a new state-of-the-art , since it improves on the best previous result of 6.0 on the MusDB test set obtained by training with 800 additional songs . We discuss in more detail the related work in the next Section . We then describe the original ConvTasnet model of Luo & Mesgarani ( 2018 ) and its adaptation to music source separation . Our Demucs architecture is detailed in Section 4 . We present the experimental protocol in Section 5 , and the experimental results compared to the state-of-the-art in Section 6 . Finally , we describe the results of the human evaluation and the ablation study . 2 RELATED WORK . A first category of methods for supervised music source separation work on time-frequency representations . They predict a power spectrogram for each source and reuse the phase from the input mixture to synthesise individual waveforms . Traditional methods have mostly focused on blind ( unsupervised ) source separation . Non-negative matrix factorization techniques ( Smaragdis et al. , 2014 ) model the power spectrum as a weighted sum of a learnt spectral dictionary , whose elements are grouped into individual sources . Independent component analysis ( Hyvärinen et al. , 2004 ) relies on independence assumptions and multiple microphones to separate the sources . Learning a soft/binary mask over power spectrograms has been done using either HMM-based prediction ( Roweis , 2001 ) or segmentation techniques ( Bach & Jordan , 2005 ) . With the development of deep learning , fully supervised methods have gained momentum . Initial work was performed on speech source separation ( Grais et al. , 2014 ) , followed by works on music using simple fully connected networks over few spectrogram frames ( Uhlich et al. , 2015 ) , LSTMs ( Uhlich et al. , 2017 ) , or multi scale convolutional/recurrent networks ( Liu & Yang , 2018 ; Takahashi & Mitsufuji , 2017 ) . Nugraha et al . ( 2016 ) showed that Wiener filtering is an efficient post-processing step for spectrogram-based models and it is now used by all top performing models in this category . Those methods have performed the best during the last SiSec 2018 evaluation ( Stöter et al. , 2018 ) for source separation on the MusDB ( Rafii et al. , 2017 ) dataset . After the evaluation , a reproducible baseline called Open Unmix has been released by Stöter et al . ( 2019 ) and matches the top submissions trained only on MusDB . MMDenseLSTM , a model proposed by Takahashi et al . ( 2018 ) and trained on 807 unreleased songs currently holds the absolute record of SDR in the SiSec campaign . Both Demucs and Conv-Tasnet obtain significantly higher SDR . More recently , models operating in the waveform domain have been developed , so far with worse performance than those operating in the spectrogram domain . A convolutional network with a U-Net structure called Wave-U-Net was used first on spectrograms ( Jansson et al. , 2017 ) and then adapted to the waveform domain ( Stoller et al. , 2018 ) . Wave-U-Net was submitted to the SiSec 2018 evaluation campaign with a performance inferior to that of most spectrogram domain models by a large margin . A Wavenet-inspired , although using a regression loss and not auto-regressive , was first used for speech denoising ( Rethage et al. , 2018 ) and then adapted to source separation ( Lluís et al. , 2018 ) . Our model significantly outperforms Wave-U-Net.Given that the Wavenet inspired model performed worse than Wave-U-Net , we did not consider it for comparison . In the field of monophonic speech source separation , spectrogram masking methods have enjoyed good performance ( Kolbæk et al. , 2017 ; Isik et al. , 2016 ) . Luo & Mesgarani ( 2018 ) developed a waveform domain methods using masking over a learnable front-end obtained from a LSTM that reached the same accuracy . Improvements were obtained by Wang et al . ( 2018 ) for spectrogram methods using the unfolding of a few iterations of a phase reconstruction algorithm in the training loss . In the mean time , Luo & Mesgarani ( 2019 ) refined their approach , replacing the LSTM with a superposition of dilated convolutions , which improved the SDR and definitely surpassed spectrogram based approaches , including oracles that use the ground truth sources such as the ideal ratio mask ( IRM ) or the ideal binary mask ( IBM ) . We show in this paper that Conv-Tasnet also outperforms all known methods for music source separation . However it suffers from significantly more artifacts than the Demucs architecture we introduce in this paper , as measured by mean opinion score . 3 ADAPTING CONV-TASNET FOR MUSIC SOURCE SEPARATION . We describe in this section the Conv-Tasnet architecture of Luo & Mesgarani ( 2018 ) and give the details of how we adapted the architecture to fit the setting of the MusDB dataset . Overall framework Each source s is represented by a waveform xs ∈ RC , T where C is the number of channels ( 1 for mono , 2 for stereo ) and T the number of samples of the waveform . The mixture ( i.e. , music track ) is the sum of all sources x : = ∑S s=1 xs . We aim at training a model g parameterized by θ , such that g ( x ) = ( gs ( x ; θ ) ) Ss=1 , where gs ( x ; θ ) is the predicted waveform for source s given x , that minimizes min θ ∑ x∈D S∑ s=1 L ( gs ( x ; θ ) , xs ) ( 1 ) for some dataset D and reconstruction error L. The original Conv-Tasnet was trained using a loss called scale-invariant source-to-noise ratio ( SI-SNR ) , similar to the SDR loss described in Section 5 . We instead use a simple L1 loss between the estimated and ground truth sources . We discuss in more details regression losses in the context of our Demucs architecture in Section 4.2 . The original Conv-Tasnet architecture Conv-Tasnet ( Luo & Mesgarani , 2018 ) is composed of a learnt front-end that transforms back and forth between the input monophonic mixture waveform sampled at 8 kHz and a 128 channels over-complete representation sampled at 1 kHz using a convolution as the encoder and a transposed convolution as the decoder , both with a kernel size of 16 and stride of 8 . The high dimensional representation is masked through a separation network composed of stacked residual blocks . Each block is composed of a a 1x1 convolution , a PReLU ( He et al. , 2015b ) non linearity , a layer wise normalization over all channels jointly ( Ba et al. , 2016 ) , a depth-wise separable convolution ( Chollet , 2017 ; Howard et al. , 2017 ) with a kernel size of 3 , a stride of 1 and a dilation of 2nmodN , with n the 0-based index of the block and N an hyper-parameter , and another PReLU and normalization . The output of each block participates to the final mask estimation through a skip connection , preceded by a 1x1 convolution . The original Conv-Tasnet counted 3×N blocks with N = 8 . The mask is obtained summing the output of all blocks and then applying ReLU . The output of the encoder is multiplied by the mask and before going through the decoder . Conv-Tasnet for music source separation We adapted their architecture to the task of stereophonic music source separation : the original Conv-Tasnet has a receptive field of 1.5 seconds for audio sampled at 8 kHz , we take N = 10 and increased the kernel size ( resp . stride ) of the encoder/decoder from 16 ( resp . 8 ) to 20 ( resp . 10 ) , leading to the same receptive field at 44.1 kHz . We observed better results using 4×N blocks instead of 3×N and 256 channels for the encoder/decoder instead of 128 . With those changes , Conv-Tasnet obtained state-of-the-art performance on the MusDB dataset , surpassing all known spectrogram based methods by a large margin as shown in Section 6 . Separating entire songs The original Conv-Tasnet model was designed for short sentences of a few seconds at most . When evaluating it on an entire track , we obtained the best performance by first splitting the input track into chunks of 8 seconds each . We believe this is because of the global layer normalization . During training , only small audio extracts are given , so that a quiet part or a loud part would be scaled back to an average volume . However , when using entire songs as input , it will most likely contain both quiet and loud parts . The normalization will not map both to the same volume , leading to a difference between training and evaluation . We did not observe any side effects when going from one chunk to the next , so we did not look into fancier overlap-add methods .

Summary: In this work, the authors consider the task of supervised music source separation, i.e., separating out the components (bass, drums, voice, other) out of a mixed music track. In particular, the work considers the task of supervised source separation where the individual target tracks are available during training time. The main contribution of this work is the improvement of an end-to-end waveform-to-waveform separation models through a number of architectural changes that allow such waveform-to-waveform models to perform comparably with other current state-of-the-art methods that instead operate in the spectrogram domain.