Discriminative and Reconstructive Basis Training for Audio Source
Separation with Semi-Supervised Nonnegative Matrix Factorization

Daichi Kitamura (SOKENDAI), Nobutaka Ono (NII/SOKENDAI),
Hiroshi Saruwatari (The University of Tokyo), Yu Takahashi (YAMAHA), Kazunobu Kondo (YAMAHA)
1. Abstract

3. How Different from Conventional Methods?

• Discriminative FSNMF [3, 4]
Separated sources

Utilizes the mixture of training signals for obtaining discriminative bases
Ex. two-source caseOracle activation for
Oracle activation forTraining stage

Spectral patterns
and their gainsArbitrary regularizer

Activations

Bases• Full-Supervised and Semi-Supervised NMF [2]
FSNMF

SSNMF

Training stage

Separation stage

Training stage,Supervised bases forSeparation stage

Fix trained bases

Should be DISCRIMINATIVE!

Fix trained bases

Should be DISCRIMINATIVE!

If the trained bases have similar patterns, the separation quality degrades.
Discriminative bases training for FSNMF is already proposed [3, 4],
but no one investigates the discriminative NMF for SSNMF.

The aim of this presentation!

2. Proposed Discriminative SSNMF

FrequencyFrequency

Frequency

Frequency

Amp. Amp. Amp.

Amp.

Unique componentof target source

Amp. Amp. Amp.

Frequency

Mixture signal

Mixture signal

Frequency

Discriminative bases

TimeTime
Time

Frequency
Amp.

Time

Spectral overlaps

Estimate correct activations with discriminative bases,
then reconstruct the target source using reconstructivebases• Optimization algorithm

We prepare a simulative mixturebases .
Training stage

for training the discriminative

Forces the other bases in
Separation stage• Simulative non-target source

The simulative non-target sourceis just possible candidatesand do not have to be the same as the non-target source in the mixture.
This signal is only used for learning- which frequency component tends to be overlapped, and- which spectra can be considered as a unique component
for the target source.
The components that never overlap with other sources- Inharmonic frequency components in piano tones- Specific spectral peaks or notches in higher frequency
Fundamental frequency peak is always overlapped (not unique).

2. Be discriminative fromthe other source spectra
as much as possible

to be different from the supervised bases in
Orthogonality regularizer.

Still the supervised bases can represent the other source components.
The discriminative supervised bases must be trained in the training
stage even in a semi-supervised context.

4. Experiments• Simple experiment using piano and flute tones

Confirm the discriminative bases obtained by the proposed method
20Overlap

-60

-100
020

Pf. (C5)

123
Frequency [kHz]45

Fl. (C6)

-20
-60

-100
020

Pf. (C5)-20

123
Frequency [kHz]Discriminative bases include somenotches at the overlap frequencies

45-20

Spectral notches-60-100020

123Frequency [kHz]45123
Frequency [kHz]45123
Frequency [kHz]45-20-60-100
020-20-60-1000• Music source separation using discriminative SSNMF
ID
123
4
5
6

Song name of mixture“Roads”“Roads”“Que pena tanto faz”

Target source

Acoustic guitar
Drums

Classic guitar“Que pena tanto faz”

Female vocals“Ultimate NZ tour”

Synthesizer“Ultimate NZ tour”

Electric guitar

Non-target source

Simulative non-target source

Drums

Drums from “The ones we love”

Acoustic guitar

Acoustic guitar from “The ones we love”

Female vocals

Male vocals from “Remember the name”

Synthesizer

Synthetic violins from “Remember the name”

Classic guitar

Electric guitar

Cross-validation for preparing… (1)• What does unique component mean?

• Penalized SSNMF [5]Fl. (C6)Time

They are conflictive!

1. Strictly represents thespectral patterns oftarget source

Separated signal

Amp.

2. Be discriminative fromthe other bases asmuch as possible

We propose to independently estimate two types of supervised bases.
Two supervised basis matrix for the target source:
Reconstructive bases
Discriminative bases

Reconstructive bases
Frequency

Proposed separation with discriminative bases

Amp.

Amp.

Time

Frequency

Time

Frequency

Time

Separated signal

Frequency

Amp. Amp.

Amp. Amp.

Amp.

Frequency

Frequency

Frequency

Mixture signal

Red: target source, Blue: non-target source
Supervised bases
Mixture signal

Frequency

Supervised bases

th source

1. Strictly represents thespectral patterns ofth source

Since the supervised bases independently (isolatedly) trained, there is noguarantee to be discriminative each other.

Conventional separation

Bilevel optimization!

This optimization finds the basis matrices by taking into account thereconstruction of each source,and.
The obtained bases tend to be discriminative.
They try to estimate the supervised bases that have two objectives:

Power [dB]

: # of frequency bins
: # of time frames: # of NMF bases

TimePower [dB]

AmplitudeTime

Power [dB]

Amplitude

Frequency

Frequency• Nonnegative Matrix Factorization (NMF) [1]Power [dB]

Mixture signal

TargetsourceNon-target
sourceElectric guitar from “Ultimate NZ tour”
Acoustic guitar from “Roads”

1816

Fromanother song

14

SDR improvement [dB]

Separate

Automatic music transcription,
speech separation, etc.

Power [dB]

• Audio Source Separation

1210

The discriminative bases are
better than the independently
trained bases for source
separation

References

864

2005 10 15 20 25 30 35 40 45 50
Number of iterations in (1)[1] D. D. Lee, et al., “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 401, pp. 788–791, 1999.
[2] P. Smaragdis, et al., “Supervised and semi-supervised separation of sounds from single-channel mixtures,” Proc. IndependentComponent Analysis and Signal Separation, pp. 414–421, 2007.
[3] P. Sprechmann, et al., “Supervised non-Euclidean sparse NMF via bilevel optimization with applications to speech enhancement,” Proc.
HSCMA, pp. 11–15, 2014.
[4] F. Weninger, et al., “Discriminative NMF and its application to single-channel source separation,” Proc. Interspeech, 2014.
[5] D. Kitamura, et al., “Music signal separation based on supervised nonnegative matrix factorization with orthogonality and maximumdivergence penalties,” IEICE Trans. Fundamentals, vol. E97-A, no. 5, pp. 1113–1118, 2014.