Music Signal Separation by Supervised Nonnegative
Daichi Kitamura, Hiroshi Saruwatari, Kiyohiro Shikano(Nara Institute of Science and Technology, Nara, Japan)
Overview of research

In proposed method, supervised bases composed in training process
can be deformed to adapt to the target spectra.

Training process

Adapt to the target
Supervision sound• The instrumental sounds differ according to, e.g., individual stylesof playing and the timbre individuality for each instrument.
• How do we get the supervision sound of the target instrument
recorded in CD?

Spectrogram of the solo-playedtarget signal for training

2. NMFAll entries of these matrix are nonnegative value.

: Activation matrix that involves activation
information of each basis ofTraining process: 500, Separation process: 400

Evaluation scores [9]Experimentally determined
Signal to distortion ratio (SDR: quality of extracted signal),
Source to interference ratio (SIR: degree of separation),
Sources to artifact ratio (SAR: absence of distortion)

UpdateExperimental results
TargetsoundPiano
Piano
Clarinet
Clarinet
FluteTrombone

What is the hyper parameter8?

DeformIfAmplitude [dB]

Real soundArtificial sound by MIDI

40

These spectral
differences reduce
separation performance.

2000.51.0
1.5
2.0
Frequency [kHz]

2.5

4

1050
Proposedmethod

420
SNMF

60
SNMF

ProposedmethodSNMF

Proposedmethod23

Example of spectrograms
4

subject towhereandfor all
are the weighting parameters for penalty terms.

,

Force to become mostly
different with each other
Supervised bases

represents undesired
instrumental components.
represents spectral
difference between supervisionsound and the target sound.

Deformation bases• The update rules that minimize the cost function
Update rules

are defined as follows:

432100

Other bases

4. Proposed method

123321004Time [s]Observed signal consisting of piano and trombone

4Time [s]Oracle signal of target piano signal
443210011234Time [s]Piano signal extracted by conventional method321001234Time [s]Piano signal extracted by proposed method

References

SupervisionsoundGenerate

6Cost function

3.0• In this research, we assume that we can obtain specific solo-playedinstrumental sounds, which is the target of the separation task.
• We use the target supervision sound generated by MIDI synthesizer
because we can easily generate the training sound via MIDI.

8Frequency [kHz]A mismatch between the bases trained in advance and the target
actual sound reduces the accuracy of separation.
It is impossible to provide perfect supervision or to predict more
realistic supervision in practice.

, the deformed bases
has a risk to become .

10152

Problem of supervised method

60SDR [dB]

It is possible to decrease
until 30% of the total.

20

SAR [dB]

10.

Proposed methodSDR
SIRSAR
7.4
13.0
9.1
12.3
23.5
12.7
0.6
2.4
7.3
9.4
21.5
9.7
6.4
16.6
6.9
5.4
17.0
5.8

Average scores: entry of matrix ,which probably has positive and negative values: Hyper parameter that controls the allowable range of the negative deformation ofFor the case ofsubject tofor all,where is the weighting parameter for penalty term

Conventional method
SDR
SIR
SAR
1.0
6.1
3.5
1.6
13.7
2.1
0.1
1.8
7.2
-0.5
12.8
0.04.2
14.2
4.8
0.7
12.5
1.2

OthersoundClarinet
Trombone
FluteTrombone
PianoClarinet

SIR [dB]

Deformable term

• To deform the supervised bases , deformation term
should have both
positive and negative value, unlike normal NMF decomposition.
• However, the target spectral bases after deformationmust benonnegative.
• The deformation matrix should be constructed under the following
constrains:
Constrains

Slightly different: Observed spectrogram: Basis matrix that includes frequently-appearing
spectral patterns in as column vectors

Artificial MIDI sounds of the target instruments that consists two octave
notes, which cover all notes of the target signal
Supervision bases: 100, Other bases: 50

The supervision sounds
were generated by MIDI
synthesizer.

: Basis matrix for the deformation andshares the activation matrix withTarget signal(piano): Number of frequency bins
: Number of frames: Number of bases

Supervision sound(MIDI)
Number of bases
Number of iterationsof NMF
Parameters

Frequency [kHz]

Extracted piano signal

Force to become
mostly different

Cost function-200.0

Proposed SNMF

NMF for source separation: Activation matrix that corresponds to: Activation matrix that corresponds to• Ideally,represents the target instrumental components, andrepresents other different components from the target sounds after thedecomposition.

To cope with the problem of supervised NMF, we propose advanced
supervised NMF algorithm that employs a deformable capability forthe trained spectral bases and penalty terms for making the bases fit
into the real instrumental sound.

Mixture of instruments

Supervised bases: Supervised basis matrix that involves spectral
patterns of the target instrumental sound: Basis matrix that involves residual spectral
patterns that cannot be expressed byPurpose of our research

Sample sound of piano

UpdateObserved spectrogram that consists
multiple instrumental sources㻌 Use some sample sound of the target instrumental signal in a priori training㻌 A mismatch between the spectra trained in advance and the target sound reduces the㻌 accuracy of source separation

Slightly different

Mixing two sources selected from four sources with the input SNR of 0dB

Frequency [kHz]Supervised bases• Supervised NMF (SNMF) [7], [8]Target signal

Observed signal

The target signals wererecorded with actualmusical instruments.

㻌 Sparse representation and decomposition algorithm
㻌 NMF attempts to separate instrumental sources using spectral characteristics
㻌 Many techniques have been proposed using NMF as an unsupervised method [2] ~ [6],
㻌 It is difficult to cluster the decomposed spectral patterns into a specific target soundSupervision soundFlute, Clari
Clarinet,t, Piano,
Pia
Trombone
Tr bo

Separation process

In addition, to prevent the simultaneous formulation ofsimilar spectral patterns in the matrices and , aspecific penalty is imposed between and .

n Previous research• Nonnegative matrix factorization (NMF) [1]Target instruments

Target signals• Recently, music signal separation technologies have received muchattention.
Applications• Automatic music transcription• Sound augmented reality (AR)• 3D audio system, etc.

Fix trained bases
and update• To confirm the effectiveness of the proposed algorithm, we compared theconventional method (SNMF) and our SNMF with basis deformation.
Experimental condition

Frequency [kHz]1. Introduction

Target soundProposed SNMF with basis deformationn Decomposition model• We employed a deformable term
to adapt the supervision bases into thetarget sound that cannot be represented by .

Separation process

To solve this inherent problem…
n We propose a new advanced supervised method that adapts thesupervision sound (that we can get easily) to the target instrumental
sound as “Supervised NMF with basis deformation.”n From experimental results, our proposed method separated the real
instrumental sources using supervision sound generated by MIDI
synthesizer.

5. Evaluation experimentStrategyPenalized Supervised NMF: PSNMF [8]Problem of conventional supervised method

Proposed SNMF adapts supervisionsound to the target signal

Kazunobu Kondo, Yu Takahashi(Yamaha Corporate Research & Development Center, Shizuoka, Japan)

3. Conventional method

n We address a music signal separation problem, and propose a newsupervised algorithm for real instrumental signal separation.
n Nonnegative matrix factorization (NMF) is a sparse representation
algorithm.
n Supervised NMF can extract the target instrumental sources.

Extract!

Matrix Factorization with Basis Deformation

MIDI data of the same type of target instrument• In this method, we need not prepare a lot of real sound samples of thetarget instrument to learn the variance of the instrumental sounds.

where,[1] D. D. Lee, H. S. Seung, “Algorithms for non-negative matrix factorization,” Neural Info. Process. Syst., vol.13, pp.556–562, 2001.
[2] T. Virtanen, “Monaural sound source separation by nonnegative matrix factorization with temporal continuity and sparseness criteria,”
IEEE Trans. Audio, Speech and Language Processing, vol.15, pp.1066–1074, 2007.
[3] S. A. Raczynski, N. Ono, S. Sagayama, “Multipitch analysis with harmonic nonnegative matrix approximation,” Proc. ISMIR, pp.381–386,
2007.
[4] A. Ozerov, C. Fevotte, M. Charbit, “Factorial scaled hidden Markov model for polyphonic audio representation and source separation,”
Proc. WASPAA, 2009.
[5] W. Wang, A. Cichocki, J. A. Chambers, “A multiplicative algorithm for convolutive non-negative matrix factorization based on squared
Euclidean distance,” IEEE Trans. on Signal Processing, vol.57, no.7, pp.2858–2864, 2009.
[6] H. Kameoka, M. Nakano, K. Ochiai, Y. Imoto, K. Kashino, S. Sagayama, “Constrained and regularized variants of non-negative matrix
factorization incorporating music-specific constraints,” Proc. ICASSP, pp.5365–5368, 2012.
[7] P. Smaragdis, B. Raj, M. Shashanka, “Supervised and semi-supervised separation of sounds from single-channel mixtures,” Proc.
LVA/ICA 2010, LNCS 6365, pp.140–148, 2010.
[8] K. Yagi, Y. Takahashi, H. Saruwatari, K. Shikano, K. Kondo, “Music signal separation by orthogonality and maximum-distance constrained
nonnegative matrix factorization with target signal information,” Proc. Audio Engineering Society 45th International Conference, 2012.
[9] E. Vincent, R. Gribonval, C. Fevotte. “Performance measurement in blind audio source separation,” IEEE Trans. Audio, Speech andLanguage Processing, vol.14, no.4, pp.1462–1469, 2006.