Music Source Separation
FREQUENCY DOMAIN: D3NET, TAKAHASHI+, CVPR2021
WAVEFORM DOMAIN: DEMUCS, DEFOSSEZ+, ARXIV2021
Nabarun Goswami
Department of Advanced Interdisciplinary
Studies, The University of Tokyo, D1

Music Production or Mixing

Image Source: https://source-separation.github.io/tutorial/_images/mixing_overview.png
Audio Source: Al James - Schoolboy Facination from test set of MUSDB18-7s dataset

Music Source Separation or De-mixing

Image Source: https://source-separation.github.io/tutorial/_images/source_separation_io.png

Sounds are just a bunch of Sine Waves

G (196Hz) violin

Images Source: https://medium.com/age-of-awareness/the-science-behind-the-sound-10bdc94ad70

Time-Domain Representation of Audio
Digital audio is just a sequence of amplitude values sampled at a certain rate.

Image Source: https://jvbalen.github.io/notes/waveform.html

Image source: https://mp4gain.com/mp4gain/wp-content/uploads/2019/09/sample-rate-0.jpg

Typical Sampling Rates:- 8kHz: walkie talkie/telephone- 16kHz : VoIP
- 44.1kHz : CD Music- 96kHz : DVD/BluRay

Time-Frequency Representation of Audio
Image Source: https://sourceseparation.github.io/tutorial/_images/right_represent
ation.png

Image Source:
https://www.geogebra.org/resource/hwtd5jkk/dvdK7
uhml3NL7tKh/material-hwtd5jkk.png

Fourier Transform of a Square Wave

Short-time Fourier Transform

Image source: https://source-separation.github.io/tutorial/_images/stft_process.png

Two ways of Representing Sounds
Two ways of Tackling Source Separation
Frequency Domain

Time Domain or Waveform Domain

MixtureVocalsDrums

MixtureBass
Vocals

Bass

Drums

Other

Other

D3NetDensely connected multidilated convolutional networks for dense prediction tasks(CVPR 2021, Takahashi+)

Image Source: https://openaccess.thecvf.com/content/CVPR2021/papers/Takahashi_Densely_Connected_Multi-Dilated_Convolutional_Networks_for_Dense_Prediction_Tasks_CVPR_2021_paper.pdf

D3NetMusic Source Separation in Frequency Domain
Multi-dilated Convolutions has anti aliasing property
Aliasing

Image Source: https://support.ircam.fr/docs/AudioSculpt/3.0/res/aliasing.png

Image Source: https://openaccess.thecvf.com/content/CVPR2021/papers/Takahashi_Densely_Connected_Multi-Dilated_Convolutional_Networks_for_Dense_Prediction_Tasks_CVPR_2021_paper.pdf

Demucs
Music Source Separation in the Waveform Domain(Defossez+, Preprint, 2021)

Image Source: https://hal.archives-ouvertes.fr/hal-02379796/document

iSeparate
Implementation and Reproduction of Music Source Separation Methods
https://github.com/media-comp/2022-iSeparate

Implementation Details❖

Framework: Pytorch❖

Training Dataset: MUSDB18▪

“The musdb18 is a dataset of 150 full lengths music tracks (~10h duration) of different genres along with theirisolated drums, bass, vocals and others stems.” (https://sigsep.github.io/datasets/musdb.html)❖Infrastructure:▪

4x Nvidia A100 GPU with 80GB VRAM (not all 80GB was used)▪

Batch size: 32 per GPU▪

Automatic Mixed Precision❖ Training Time:▪

D3Net (vocals): ~0.5 days▪

Demucs (all sources): ~4 days❖

Model Size on disk:▪

D3Net (vocals): ~13 MB▪

Demucs (all sources): ~1 GB

Results from Current Implementation

Audio files are too big to embed in the PowerPoint presentation.
Demo in external Audio software.

Pull Requests➢ Train D3Net for other sources➢ Verify Demucs implementation (results are not as good as reported in the paper or the official code)➢ Create a web-app (maybe a huggingface space)➢ Create a VST plugin for integration in Digital Audio Workstations (https://audioassemble.com/how-to-make-a-vst/)➢ Bugs and Fixes➢ Implement other methods➢ Anything else…

Thank you