Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Beyond NMF:Time-Domain
Audio Source Separation
without Phase Reconstruction
PSDTF

Kazuyoshi Yoshii (AIST, Japan)
Ryota Tomioka(Univ. of Tokyo , Japan)
Daichi Mochihashi (ISM , Japan)
Masataka Goto
(AIST , Japan)

2013/11/06 ISMIR 2013

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Summary

If you do not care about
computational time,
you could forget NMF anduse our method PSDTFfor audio source separation
We provide MATLAB source codes!

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Overview•

Our goal: high-quality source separation of monaural audio signals

Time-domain separation
is equivalent tofrequency-domain separation
Time domain
MixturePowertime

Separate
Source 1
Source 2
Source 3

Frequency domain
freq

Equiv.

Phase
freq

SeparateBeyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Overview•Our goal: high-quality source separation of monaural audio signals– Nonnegative matrix factorization (NMF) is very popular• Frequency-domain decomposition for power spectrogramTime domain

MixturePowertime

Separate
Source 1

Frequency domain
freq

Separate

Equiv.

Source 2
Source 3

NMF

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Overview•Our goal: high-quality source separation of monaural audio signals– Nonnegative matrix factorization (NMF) is very popular• Frequency-domain decomposition for power spectrogram• Ignore phase → Cannot separate time-domain signals wellTime domain

MixturePowertime

Separate
Source 1

Frequency domain
freq

freq

Separate

Equiv.

Source 2
Source 3

NMFPhaseBeyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Overview•

Our goal: high-quality source separation of monaural audio signals

Our PSDTF can solve this problem
by time-domain separation!
Time domain
MixturePower

Frequency domain

time

Separate
Source 1

freq

Source 2Source 3

Time-domain PSDTF

freq

Separate

Equiv.

NMF

Phase

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Overview•Our goal: high-quality source separation of monaural audio signals

Time-domain PSDTF has an equivalent form of
frequency-domain PSDTF considering the phase
Time domain
MixturePowerFrequency domain

time

Separate
Source 1

freq

Phase
freq

Separate

Equiv.

Source 2
Source 3

Time-domain PSDTF

NMF

Freq-domain PSDTF

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Nonnegative Matrix Factorization (NMF)•A nonnegative matrix (power spectrogram) is approximatedby the product of two nonnegative matrices– Given the observation matrix , NMF estimatesandso that their productcan be similar toReconstruction

Basis matrix

frequency

Observation

IS
time

Itakura-Saito (IS) divergence
is often used as a cost function

Activation
matrixBeyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Nonnegative Matrix Factorization (NMF)•

Nonnegative vectors are approximatedby convex combinations of fewer nonnegative vectors
time

…

…

…

…

Nonnegativevectors… Observations

A series of nonnegative observed vectors along the time
Each vector corresponds to the power spectrogram ofa locally observed mixture signal at each frame

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Nonnegative Matrix Factorization (NMF)•

Nonnegative vectors are approximatedby convex combinations of fewer nonnegative vectors
time

Nonnegativevectors

…

…

…

…

Nonnegativevectors… Observations

Bases

A set of nonnegative basis vectorsthat usually correspond to different pitches
Each observed vector is approximatedby a weighted sum of these basis vectors

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Nonnegative Matrix Factorization (NMF)•

Nonnegative vectors are approximatedby convex combinations of fewer nonnegative vectors
time

Nonnegativevectors

…

…

…

…

Nonnegativevectors… Observations

Bases

Such weights arecalled activations
Activations

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Nonnegative Matrix Factorization (NMF)• goalNonnegative
are approximated
Theof NMFvectors
is to estimate
basis and activation vectors
byso
convexof fewer nonnegativevectorsthat combinations
the total reconstruction
error canbe minimized
time

Nonnegative
vectors

…

…

…

Bases

…

Nonnegative
vectors… Observations

Power spectrum
IS divergenceis minimized

Activations

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Positive SemiDefinite Tensor Factorization (PSDTF)
Positive semidefinite (PSD) matrices are approximatedby convex combinations of fewer PSD matrices
time

…

…

…

…

Local signal•

Local signal

Outer prod.

PSD matrices… Observations

Covariance

The observed PSD matrices are a set of local covariance matrices
calculated from a locally observed mixture signal at each frame
By calculating the outer product of the local signal,
we can make the PSD matrix

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Positive SemiDefinite Tensor Factorization (PSDTF)
Positive semidefinite (PSD) matrices are approximatedby convex combinations of fewer PSD matrices
time

PSD matrices

Bases

…

…

…

…

Local signal•Local signal

Outer prod.

PSD matrices… Observations

Covariance

A set of basis matrices (PSD matrices)
Each basis matrix represents statisticalcharacteristics of the source signal
Each observed PSD matrix is approximatedby a weighted sum of these basis matrices

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Positive SemiDefinite Tensor Factorization (PSDTF)
Positive semidefinite (PSD) matrices are approximatedby convex combinations of fewer PSD matrices
time

PSD matrices

Bases

…

…

…

…

Local signal•Local signal

Outer prod.

PSD matrices… Observations

Covariance

Such weights arecalled activations(same with NMF)

Activations

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Positive SemiDefinite Tensor Factorization (PSDTF)
Local signal

Local signal• goalPositivesemidefinitematrices
arematrices
approximated
Theof PSDTF
is to (PSD)
estimatebasis
and activationby convex
of fewer PSD matrices
vectors
so thatcombinations
the total reconstruction
error can be minimizedtime

PSD matrices

Bases

…

…

…

…

Outer prod.

PSD matrices… Observations

Covariance
Log-Det divergenceis minimized

Activations

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Nonnegative Matrix Factorization (NMF)•

Nonnegative vectors are approximatedby convex combinations of fewer nonnegative vectors
time

Nonnegativevectors

…

…

…

Bases

…

Nonnegative
vectors… Observations

Power spectrum
IS divergenceis minimized

Activations

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Positive SemiDefinite Tensor Factorization (PSDTF)
Positive semidefinite (PSD) matrices are approximatedby convex combinations of fewer PSD matrices
time

PSD matrices

Bases

…

…

…

…

Local signal•Local signal

Outer prod.

PSD matrices… Observations

Covariance
Log-Det divergenceis minimizedActivations

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on NMF•

Wiener filtering based on NMF results– Element-wise decomposition of nonnegative power spectrogram
MixtureSource 1

Source 2

Source 3

PowerAs shown in this equation, Wiener filtering can be regarded
as a simple power partition functionThe power of each element
is distributed into
different sources,,Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on NMF•

Wiener filtering based on NMF results– Element-wise decomposition of nonnegative power spectrogram– Ignore phases and treat frequency bins independently
MixtureSource 1

Source 2

Source 3

PowerPhase

Just copy & pasteBeyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on NMF•

Wiener filtering based on NMF results– Element-wise decomposition of nonnegative power spectrogram– Ignore phases and treat frequency bins independently
MixtureSource 1

Source 2

Source 3

PowerPhase

Just copy & pasteBeyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on PSDTF•

Wiener filtering based on PSDTF results– Vector-wise decomposition of complex spectrogram
MixtureComplex spectrogram

PowerPhase

Source 1

Source 2

Source 3Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on PSDTF•

Wiener filtering based on PSDTF results

The –complexspectrogramof theofobserved
mixture signal is
Vector-wisedecomposition
complex spectrogram
decomposed in a vector-wise/frame-wise manner unlike NMF
MixtureComplex spectrogram

PowerPhase

Source 1

Source 2

Source 3

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on PSDTF•Wiener filtering based on PSDTF results– Vector-wise decomposition of complex spectrogram– Consider phases and correlations between frequency bins
MixtureSource 1

Source 2

Source 3

Complex spectrogram

PowerPhase

Estimate phases!

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on PSDTF•

Wiener filtering based on PSDTF results

PSDTF
considers thecorrelations
frequency bins!
– Vector-wisedecomposition
ofbetweencomplex spectrogram
Harmonicstructure
hasstrongcorrelations
betweenharmonics– Considerphases
andcorrelations
betweenfrequencybins
MixtureSource 1

Source 2

Source 3

PowerPhase

Estimate phases!

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on PSDTF•

Wiener filtering based on PSDTF results– Vector-wise decomposition of complex spectrogram– Consider phases and correlations between frequency bins
MixtureSource 1

Source 2

Source 3

PowerPhase

Estimate phases!

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on NMF•

Wiener filtering based on NMF results– Element-wise decomposition of nonnegative power spectrogram– Ignore phases and treat frequency bins independently
MixtureSource 1

Source 2

Source 3

PowerPhase

Just copy & pasteBeyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Source Separation based on PSDTF•

Wiener filtering based on PSDTF results– Vector-wise decomposition of complex spectrogram– Consider phases and correlations between frequency bins
MixtureSource 1

Source 2

Source 3

PowerPhase

Estimate phases!

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

PSD Matrices

I intentionally skippeda very important explanation

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Positive SemiDefinite Tensor Factorization (PSDTF)
Positive semidefinite (PSD) matrices are approximatedby convex combinations of fewer PSD matrices
time

PSD matrices

Bases

…

Local signal•Local signal

Outer prod.

PSD matrices

What arerepresented by
PSD matrices?
…

…

…

… Observations

Covariance

Log-Det divergenceis minimizedActivations

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Gaussian Process (GP)•A stochastic process over continuous time
– GP can be interpreted as a probability distribution
over continuous functions:→

PSD matrices

Bases

PSDTF matrix isthe kernel matrix (covariance matrix)of Gaussian Process (GP)

Kernel matrix(PSD covariance matrix)

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Gaussian Process (GP)•A stochastic process over continuous time
– GP can be interpreted as a probability distribution
over continuous functions:→

Three functions are stochastically sampled from the same GP

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Gaussian Process (GP)•A stochastic process over continuous time
– GP can be interpreted as a probability distribution
over continuous functions:

The “likelihood” of any signal (function) can be evaluated by GP→

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

How to Understand GP?
•Gaussian process is characterized by its marginal distribution– A marginal distribution over anypoints can be Gaussian

Analogue wave: Gaussian process
Digital wave: Gaussian distribution

Kernel matrix
The function dynamics are determined(PSD covariance matrix)
by the kernel matrix

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

How to Understand GP?
•Gaussian process is characterized by its marginal distribution– A marginal distribution over any

points can be Gaussian

Analogue wave: Gaussian process
Digital wave: Gaussian distribution

This means that the uncertainty over is modeled bya mean vector and a covariance matrix (kernel matrix)

Kernel matrix
The function dynamics are determined(PSD covariance matrix)
by the kernel matrix

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Example: Identity Kernel•

Stationary Gaussian white noise can be generated– Marginal:In this kernel,
there is no correlation
between samples→ Its GP can generate
white noise

We consider equally-spanned sampling points

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Example: Squared Exponential Kernel (RBF)•

Smooth functions can be generated– Marginal:

Strong correlation in the vicinity→ Slowly changing function

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Example: Periodic Kernel (Stripe Pattern)•Periodic functions can be generated– Marginal:

Strong corr. at a certain interval→ Periodic function

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Positive SemiDefinite Tensor Factorization (PSDTF)
Positive semidefinite (PSD) matrices are approximatedby convex combinations of fewer PSD matrices
time

PSD matrices

Bases

…

…

…

…

Local signal•

Local signal

Outer prod.

PSD matrices… Observations

Covariance
Log-Det divergenceis minimizedActivations

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

NMF vs. PSDTF•

Nonnegative Matrix Factorization (NMF)– Nonnegative vectors → sums of nonnegative vectors– Bregman divergence• Itakura-Saito (IS) divergence• Kullback-Leibler (KL) divergence•

Positive Semidefinite Tensor Factorization (PSDTF)– PSD matrices → sums of PSD matrices– Bregman “matrix” divergence• Log-Determinant (LD) divergence• von Neumann (vN) divergenceBeyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Frequency-domain PSDTF•

PSDTF is a natural extension of NMF

PSDTF

NMF

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Comparative Evaluation•Source separation performance on synthetic sounds– 3 bases (K=3)– 3 instruments (from RWC Music Database)• Piano / Electric guitar (decaying) / Clarinet (sustaining)– BSS Eval Toolbox [Vincent2006] was used
C4E4

G4C+EC+GE+GC+E+G

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Experimental Results•

LD-PSDTF outperformed KL-NMF and IS-NMF[dB] 30

Signal-to-Distortion
Ratio (SDR)

25

Signal-to-Interferences
Ratio (SIR)

20

Signal-to-Artifacts
Ratio (SAR)

15
10SDR

SIR

SAR

KL-NMF
IS-NMF
LD-PSDTF (proposed)

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)Conclusion and Future Work•

Positive Semidefinite Tensor Factorization (PSDTF)– A natural extension of Nonnegative Matrix Factorization (NMF)– Nonparametric Bayesian extension [Yoshii et al. ICML 2013]• Automatically optimize the number of bases

If you do not care about
computational time,
you could forget NMF anduse our method PSDTF
for audio source separation

We provide MATLAB source codes! (2-Clause BSD License)
http://staff.aist.go.jp/k.yoshii/psdtf/
or the last tweet of @MasatakaGoto

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

Beyond NMF:Time-Domain
Audio Source Separation
without Phase Reconstruction
PSDTF

Kazuyoshi Yoshii (AIST, Japan)
Ryota Tomioka(Univ. of Tokyo , Japan)
Daichi Mochihashi (ISM , Japan)
Masataka Goto
(AIST , Japan)

2013/11/06 ISMIR 2013

Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction (Yoshii, Tomioka, Mochihashi, Goto)

We Cannot Ignore Computational Time
• Current issue: Big computational cost!
– Comparison of computational cost• #samples: M=512(frame size: 32ms)• #frames: N=1400(duration: 14s)
• #bases:K=3(Mixtures of 3 sounds)• MATLAB / Intel Xeon 3.4GHz• LD-PSDTF: 600s/iteration– 16.7 hours (60000s) / 100iterations• IS-NMF: 0.1s/iteration– 10s / 100iterations