Package: xumx-spleeterweb
CrossNet-Open-Unmix (X-UMX) for Spleeter WebThis is a modified version of theofficial X-UMX repomade to be compatible withSpleeter Web!This repository contains the NNabla implementation ofCrossNet-Open-Unmix (X-UMX), an improved version ofOpen-Unmix (UMX)for music source separation. X-UMX achieves an improved performance without additional learnable parameters compared to the original UMX model. Details of X-UMX can be found inour paper.Quick Music Source Separation Demo by X-UMXFrom the Colab link below, you can try using X-UMX to generate and listen to separated audio sources of your audio music file. Please give it a try!Related Projects:x-umx |open-unmix-nnabla|open-unmix-pytorch|musdb|museval|norbertThe ModelAs shown in Figure (b),X-UMXhas almost the same architecture as the original UMX,
but only differs by two additional average operations that link the instrument models together.
Since these operations are not DNN layers, the number of learnable parameters ofX-UMXis
the same as for the original UMX and also the computational complexity is almost the same.
Besides the model, there are two more differences compared to the original UMX. In particular,
Multi Domain Loss (MDL) and a Combination Loss (CL) are used during training, which are different
from the original loss function of UMX. Hence, these three contributions, i.e., (i) Crossing architecture,
(ii) MDL and (iii) CL, make the original UMX more effective and successful without additional learnable parameters.Getting startedInstallationFor installation we recommend to use theAnacondapython distribution. To create a conda environment foropen-unmix, simply run:conda env create -f environment-X.ymlwhereXis either [cpu,gpu], depending on your system.Source separation with pretrained modelHow to separate using pre-trained X-UMXDownloadherea pre-trained model of X-UMX which results in the scores given inour paper.
The model was trained on theMUSDB18dataset.In order to use it, please use the following command:python-mxumx.test--inputs[Inputmixture(anyaudioformatsupportedbyFFMPEG)]--model{pathtodownloadedx-umx.h5weightsfile}--contextcpu--chunk-dur10--outdir./results/Please note that our X-UMX integrates the different instrument networks of the original UMX by a crossing operation, and thus X-UMX requires more memory. So, it maybe difficult to run the model on smaller GPU. So, though default choice is GPU inference, above example uses the option--context cpu. Also note that because memory requirement is high, we suggest users to set--chunk-durwith values appropriate for each computer. It is used to break audio into smaller chunks, separate sources and stitch them back together. If your inference crashes, kindly reduce chunk duration and try again.Evaluation usingmusevalTo perform evaluation in comparison to other SiSEC systems, you would need to install themusevalpackage usingpip install musevaland then run the evaluation usingpython-mxums.eval--model[pathtodownloadedx-umx.h5model]--root[PathofMUSDB18]--outdir[Pathtosavemusdbestimates]--evaldir[Pathtosavemusevalresults]Training X-UMXX-UMX can be trained using the default parameters of the train.py function.TheMUSDB18is one of the largest freely available datasets for professionally produced music tracks (~10h duration) of different styles. It comes with isolateddrums,bass,vocalsandothersstems.MUSDB18contains two subsets: "train", composed of 100 songs, and "test", composed of 50 songs.To directly trainx-umx, we first would need to download the dataset and place inunzippedin a directory of your choice (calledroot).ArgumentDescriptionDefault--root <str>path to root of dataset on disk.NoneAlso note that, if--rootis not specified, we automatically download a 7 second preview version of the MUSDB18 dataset. While this is comfortable for testing purposes, we wouldn't recommend to actually train your model on this.Training (Single/Distributed training) can be started using below commands.Single GPU trainingpython-mxumx.train--root[PathofMUSDB18]--output[Pathtosaveweights]Distributed TrainingFor distributed traininginstall NNabla package compatible with Multi-GPU execution. Use the below code to start the distributed training.export CUDA_VISIBLE_DEVICES=0,1,2,3 {device ids that you want to use}mpirun-n{no.ofdevices}python-mxumx.train--root[PathofMUSDB18]--output[Pathtosaveweights]Please note that above sample training scripts will work on high quality 'STEM' or low quality 'MP4 files'. In case you would like faster data loading, kindly look atmore details hereto generate decoded 'WAV' files. In that case, please use--is-wavflag for training.TrainingMUSDB18usingx-umxcomes with several design decisions that we made as part of our defaults to improve efficiency and performance:chunking: we do not feed full audio tracks intoopen-unmixbut instead chunk the audio into 6s excerpts (--seq-dur 6.0).balanced track sampling: to not create a bias for longer audio tracks we randomly yield one track from MUSDB18 and select a random chunk subsequently. In one epoch we select (on average) 64 samples from each track.source augmentation: we apply random gains between0.25and1.25to all sources before mixing. Furthermore, we randomly swap the channels the input mixture.random track mixing: for a given target we select arandom trackwith replacement. To yield a mixture we draw the interfering sources from different tracks (again with replacement) to increase generalization of the model.fixed validation split: we provide a fixed validation split of14 tracks. We evaluate on these tracks in full length instead of using chunking to have evaluation as close as possible to the actual test data.Some of the parameters for the MUSDB sampling can be controlled using the following arguments:ArgumentDescriptionDefault--is-wavloads the decoded WAVs instead of STEMS for faster data loading. Seemore details here.False--samples-per-track <int>sets the number of samples that are randomly drawn from each track64--source-augmentations <list[str]>applies augmentations to each audio source before mixinggain channelswapTraining and Model ParametersAn extensive list of additional training parameters allows researchers to quickly try out different parameterizations such as a different FFT size. The table below, we list the additional training parameters and their default values:ArgumentDescriptionDefault--output <str>path where to save the trained output model as well as checkpoints../x-umx--model <str>path to checkpoint of target model to resume training.not set--epochs <int>Number of epochs to train1000--batch-size <int>Batch size has influence on memory usage and performance of the LSTM layer16--patience <int>Early stopping patience1000--seq-dur <int>Sequence duration in seconds of chunks taken from the dataset. A value of<=0.0results in full/variable length6.0--unidirectionalchanges the bidirectional LSTM to unidirectional (for real-time applications)not set--hidden-size <int>Hidden size parameter of dense bottleneck layers512--nfft <int>STFT FFT window length in samples4096--nhop <int>STFT hop length in samples1024--lr <float>learning rate0.001--lr-decay-patience <int>learning rate decay patience for plateau scheduler80--lr-decay-gamma <float>gamma of learning rate plateau scheduler.0.3--weight-decay <float>weight decay for regularization0.00001--bandwidth <int>maximum bandwidth in Hertz processed by the LSTM. Input and Output is always full bandwidth!16000--nb-channels <int>set number of channels for model (1 for mono (spectral downmix is applied,) 2 for stereo)2--seed <int>Initial seed to set the random initialization42--valid_dur <float>To prevent GPU memory overflow, validation is calculated and averaged pervalid_durseconds.100.0AuthorsRyosuke Sawata(*), Stefan Uhlich(**), Shusuke Takahashi(*) and Yuki Mitsufuji(*)(*)  Sony Corporation, Tokyo, Japan(**)Sony Europe B.V., Stuttgart, GermanyReferencesIf you use CrossNet-open-unmix for your research – Cite CrossNet-Open-Unmix@article{sawata20,  
  title={All for One and One for All: Improving Music Separation by Bridging Networks},
  author={Ryosuke Sawata and Stefan Uhlich and Shusuke Takahashi and Yuki Mitsufuji},
  year={2020},
  eprint={2010.04228},
  archivePrefix={arXiv},
  primaryClass={eess.AS}}If you use open-unmix for your research – Cite Open-Unmix@article{stoter19,  
  author={F.-R. St\\"oter and S. Uhlich and A. Liutkus and Y. Mitsufuji},  
  title={Open-Unmix - A Reference Implementation for Music Source Separation},  
  journal={Journal of Open Source Software},  
  year=2019,
  doi ={10.21105/joss.01667},
  url ={https://doi.org/10.21105/joss.01667}}If you use the MUSDB dataset for your research - Cite the MUSDB18 Dataset@misc{MUSDB18,
  author       ={Rafii, Zafar and
                  Liutkus, Antoine and
                  Fabian-Robert St{\"o}ter and
                  Mimilakis, Stylianos Ioannis and
                  Bittner, Rachel},
  title        ={The{MUSDB18}corpus for music separation},
  month        = dec,
  year         = 2017,
  doi          ={10.5281/zenodo.1117372},
  url          ={https://doi.org/10.5281/zenodo.1117372}}If compare your results with SiSEC 2018 Participants - Cite the SiSEC 2018 LVA/ICA Paper@inproceedings{SiSEC18,
  author="St{\"o}ter, Fabian-Robert and Liutkus, Antoine and Ito, Nobutaka",
  title="The 2018 Signal Separation Evaluation Campaign",
  booktitle="Latent Variable Analysis and Signal Separation:
  14th International Conference, LVA/ICA 2018, Surrey, UK",
  year="2018",
  pages="293--305"}⚠️ Please note that the official acronym forCrossNet-Open-UnmixisX-UMX.