3-2-9
Optimal divergence diversity for superresolution-based
nonnegative matrix factorization ∗◎ Daichi Kitamura, Hiroshi Saruwatari, Satoshi Nakamura (Nara Institute of Science and Technology)
Kazunobu Kondo, Yu Takahashi (Yamaha Corporation), Hirokazu Kameoka (The University of Tokyo)

1 Introduction

equation represents the decomposition in SNMF;

In recent years, music and acoustic signal separation based on nonnegative matrix factorization (NMF)[1] has been a very active area of signal-processing research. NMF of acoustic signals decomposes an input
spectrogram into the product of a spectral basis matrix
and its activation matrix. In particular, supervised NMF(SNMF) [2], which includes a priori training with somesample sounds of a target instrument, can extract the target signal to some extent. However, in the case of a mixture consisting of many sources, the source extraction
performance is markedly degraded when only singlechannel observation is available.
Multichannel NMF, which is a natural extension of
NMF to a multichannel signal, has been proposed as anunsupervised method [3]. However, such an unsupervised separation is a diﬃcult problem because the decomposition is underspecified. Hence, algorithms used
for multichannel NMF have strong dependence on initial values and lack robustness.
As another means of addressing multichannel signal separation, a hybrid method, which concatenates
superresolution-based SNMF after directional clustering, has been proposed by the authors [4, 5]. Thismethod uses index information generated by binary
masking of directional clustering so that spectral chasms
can be regarded as unseen observations, and finally reconstructs the target source components via spectrogram extrapolation using the supervised bases as a dictionary. Also, we have proposed some update algorithms for the superresolution-based SNMF based on βdivergence, which includes Itakura-Saito divergence (ISdivergence), generalized Kullback-Leibler divergence,(KL-divergence) and Euclidean distance (EUC-distance)[5]. In general SNMF-based music signal separation,
KL-divergence is often used as a cost function because
the spectrogram of the music signals tends to become
sparse, and KL-divergence-based SNMF fits to represent such sparse signals [6]. However, it has been experimentally confirmed that the optimal divergence forsuperresolution-based SNMF is EUC-distance [5]. Thisdiﬀerence of performance between EUC-distance andKL-divergence is due to the diﬀerence of a spatial condition in each source, and the optimal divergence temporally fluctuates because the spatial condition is not consistent in the general music signal. Therefore, these divergence should be changed in each time frame automatically. To solve this problem, in this paper, we propose a
new scheme for frame-wise divergence diversity to separate the target signal using optimal divergence.

2 Conventional Method
2.1 SNMF
In SNMF, a priori spectral patterns (bases) should betrained in advance as a basis dictionary. The following∗

Y ≃ F G + HU ,(1)

where Y (∈ RΩ×T
is an observed spectrogram,≥0 )F (∈ RΩ×K≥ 0 ) is a supervised basis matrix trained inadvance, which includes spectral patterns of the target
signal as column vectors, G(∈ R≥K×T0 ) is the activation
matrix that corresponds to F , H(∈ RΩ×L
≥ 0 ) represents theresidual spectral patterns that cannot be expressed by
F G, and U (∈ R≥L×T
0 ) is the activation matrix that corresponds to H. Moreover, Ω is the number of frequency
bins, T is the number of frames of the observed signal,
K is the number of bases of F , and L is the numberof bases of H. In SNMF, the matrices G, H, andU are optimized under the condition that F is known
in advance. The matrix F can be trained by solving
Ŷ = F Ĝ, where Ŷ is a training data spectrogram andĜ is the corresponding activation matrix. Hence, F G
ideally represents the target instrumental components
and HU represents other components diﬀerent fromthe target sounds after the decomposition.
SNMF can extract the target signal, particularly in thecase of a small number of sources. However, in the caseof a mixture consisting of many sources, the source extraction performance is markedly degraded because of
the existence of similar-timbre instruments.
2.2 Directional Clustering and Its Hybrid Methodwith Superresolution-Based SNMF
Decomposition methods employing directional information for the multichannel signal have also been proposed as unsupervised techniques [7]. These methods
quantize the direction via time-frequency binary masking. Such directional clustering works well, even in anunderdetermined situation. However, there is an inherent
problem that the sources located in the same direction
cannot be separated only using directional information.
Furthermore, the separated signal is likely to be distorted
because the signal has many spectral chasms resulting
from the binary-masking procedure as shown in Fig. 1.
To solve this problem, a hybrid method that concatenates superresolution-based SNMF after directional
clustering has been proposed [4]. This SNMF algorithm
explicitly utilizes index information determined by timefrequency binary masking in directional clustering. For
example, if the target instrument is localized in the center cluster along with the interference, superresolutionbased SNMF is only applied to the existing center components using index information (see Fig. 1). Therefore,
the spectrogram of the target instrument is reconstructedusing more matched bases because spectral chasms aretreated as unseen, and these chasms have no impact on
the cost function in SNMF. In addition, the componentsof the target instrument lost after directional clustering
can be extrapolated using the supervised bases. In otherwords, the resolution of the target spectrogram is recovered with the superresolution by the supervised basis extrapolation. Furthermore, a regularization term is added“超解像型教師あり非負値行列因子分解における最適なダイバージェンスのダイバーシチ,” by 北村大地, 猿渡洋,
中村哲 (奈良先端大), 近藤多伸, 高橋祐 (ヤマハ株式会社), 亀岡弘和 (東京大).

日本音響学会講演論文集- 727 -

2014年3月

Index oftargetcluster

Spectrum separated bydirectional clustering

MixingExtracted signal

1

2

3

4

β NMF

Fig. 2 Trade-oﬀ between separation and extrapolation
abilities.
Frequency

Spectrum reconstructedby superresolution

Signal flow of conventional hybrid method.

in the cost function to avoid a basis extrapolation error[8].
2.3 Cost Function for Superresolution-Based
SNMF
Here, the index matrix I(∈ RΩ×T
{0, 1} ) is obtained fromthe binary masking preceding the directional clustering.
This index matrix has specific entries of unity or zero,
which indicate whether or not each grid of the spectrogram belongs to the target directional cluster. The costfunction in superresolution-based SNMF is defined using the index matrix I as [5]
∑
∑
∑
(
)
J = ω,t iω,t DβNMF yω,t ∥ k fω,k gk,t + l hω,l ul,t∑∑+ λ ω,t iω,t Dβreg (0∥ k fω,k gk,t ) + µ∥F T H∥2Fr , (2)where iω,t , yω,t , fω,k , gk,t , hω,l , and ul,t are the nonnegative
entries of matrices I, Y , F , G, H, and U , respectively,
λ and µ are the weighting parameters for each penalty
term, · represents the binary complement of each entry in
the index matrix, and ◦ indicates the Hadamard productof matrices. In addition, Dβ (·∥·) is β-divergence, which
is defined as [9]


yβ
xβ
yxβ−1



+ −
(β ∈ R\ {0, 1} ) β((β − 1) β ) (β − 1)Dβ (y∥x) = . (3)y log y − log x + x − y (β = 1)





y
y


 − log − 1(β = 0)xxThis generalized divergence is a family of cost functions parameterized by a single shape parameter β thattakes IS-divergence, KL-divergence, and EUC-distancein special cases (β = 0, 1, and 2, respectively).

3 Poposed Method
3.1 Divergence Dependency on Local Chasm Condition
In general SNMF-based music signal separation, KLdivergence is often used as a cost function because KLdivergence-based SNMF decomposes the observed spectrogram into the mixture of more sparser bases compared with EUC-distance-based SNMF. The spectrogram of the music signals tends to become sparse, andKL-divergence-based SNMF fits to represent such sparse
signals [6]. However, it has been experimentally confirmed that the optimal divergence for superresolutionbased SNMF is EUC-distance [5]. This discrepancy inthe divergence is due to the fact that superresolutionbased SNMF has two tasks, namely, signal separation
and basis extrapolation. The sparseness is not suitable

日本音響学会講演論文集

Extrapolation

0
Amplitude

ISTFT

Total performance

Separation

R-ch
Superresolutionbased SNMF

ISTFT

Fig. 1Frequency

Target clustercomponents

L-chSuperresolutionbased SNMF

Chasm

Optimum insuperresolution-based SNMF

Optimum in
SNMF

Performance

Directional clustering

Amplitude

Input stereo signal
L-ch
R-ch
STFT

for the basis extrapolation because it is diﬃcult to extrapolate sparse bases only from the observable data.
Figure 2 shows the dependency of the sparseness andseparation performance of superresolution-based SNMF.
The sparse criterion, such as KL-divergence (βNMF = 1),
is not suitable for superresolution-based SNMF.
The optimal divergence for superresolution-based
SNMF depends on the rate of spectral chasms in eachtime frame of the spectrogram obtained by directional
clustering because of the trade-oﬀ between separation and extrapolation abilities. If there are manychasms in a frame of the binary-masked spectrogram,superresolution-based SNMF is preferred to have highextrapolation ability. In contrast, if the rate of chasms
is low value, the separation ability is required rather
than the extrapolation. Therefore, it is expected that
EUC-distance should be used in the frames that have
many chasms, and KL-divergence should be used in theother frames. To improve separation performance ofsuperresolution-based SNMF for any types of input signals, we propose a new frame-wise divergence switching
method as described below.
3.2 Cost Function
Considering the above-mentioned divergence dependency on the local chasm condition, we propose toswitch the divergence in each frame of the spectrogram
according to the rate of chasms in each frame, rt , and athreshold value τ (0 ≤ τ ≤ 1), where the rate of chasms
rt can be calculated from the index matrix I. Figure 3
depicts an algorithm of the frame-wise divergence diversity. This divergence switching method is implemented
by switching the cost function in each frame, as∑
J = t Jt ,
(4)
∑
(EUC)

)

ω iω,t Dβ=2 (yω,t ∥sω,t∑∑ (EUC)



+
λ
i
D
(0∥
gk,t )
ω ω,t βregk fω,k(EUC)T2
H∥Fr(rt ≥ τ) + µ∥F
Jt = 
,
(5)
∑

(KL)

i
D
(y
∥s
)

ω,t
β=1
ω,t
ω
ω,t


∑
∑ (KL)

gk,t )+ λ ω iω,t Dβreg (0∥ k fω,k




(KL)T
2
+ µ∥F
H∥Fr
(rt < τ)
∑
∑ (∗)
(∗)
(6)
sω,t = k fω,k gk,t + n hω,n un,t ,∑rt = ( ω iω,t )/Ω,
(7)
(EUC)
where F (KL) (∈ RΩ×K(∈ RΩ×K≥ 0 ) and F≥ 0 ) are the supervised basis matrices trained in advance using KL(KL)divergence and EUC-distance, respectively. Also, fω,k(EUC)and fω,k
are the entries of F (KL) and F (EUC) , respectively, and ∗ = {KL, EUC}. The divergence is determined
depending on rt and τ in each frame. Therefore, thismethod can be considered as a frame-wise diversity ofthe divergence to achieve both of optimal separation andextrapolation.

- 728 -

2014年3月

Frequency

Table 1

Binary-masked
spectrogram

Composition
C1
C2
C3

Compositions of musical instruments
Melody 1
Oboe
TrumpetClarinet

Melody 2
FluteViolinHorn

Midrange
Piano
Harpsichord
Piano

Bass
Trombone
Fagotto
Cello

Time

Table 2
Spatialpattern
SP1
SP2
SP3
SP4

Calculation of rate of chasms
(EUC)

Yes

EUC-distancebased cost function

No

(KL)

KL-divergencebased cost function

Superresolution-based SNMF

Fig. 3 Divergence diversity algorithm of proposed
method.
3.3 Auxiliary Function and Update Rules
The update rules based on (4) is obtained by the auxiliary function approach. Similarly to [4, 5], we can design
the upper bound function J + using auxiliary variables
(∗)
(∗)
α(∗)
k,l,ω ≥ 0, γω,t,k ≥ 0, δω,t,l ≥ 0, ε1 ≥ 0, ε2 ≥ 0, and ζω,t ≥ 0
∑ (∗)
∑
∑
that satisfy ω αk,l,ω = 1, k γω,t,k = 1, l δω,t,l = 1, andε1 +ε2 = 1, as∑J ≤ J + = t Jt+ ,
(8)
)
(
∑
2


ω iω,t yω,t +vω,t +2wω,t


∑



+λ ω iω,t Rβ(EUC)


reg

( (EUC)
)

∑

2 2


hω,l /α(EUC)+  +µ (k,l,ω fω,kk,l,ω ) (rt ≥ τ)
Jt ≤ Jt = ,∑∑ (KL)


i
−y
γ
δ
Q+C

ω,t
ω,t k,l ω,t,k ω,t,l
ω


∑



+λ ω iω,t Rβ(KL)


reg

( (KL)) (KL)

2 2
 +µ ∑(rt < τ)k,l,ω fω,k hω,l /αk,l,ω
(9)
where
vω,t =∑∑(EUC) 2 2
(EUC)
gk,t )/γω,t,k+ l (hω,lul,t )/δω,t,l ,k ( fω,k(10)∑ (EUC)∑wω,t = ( k fω,k gk,t )( l hω,l ul,t ),∑ (EUC)∑− yω,t k fω,kgk,t − yω,t l hω,l ul,t ,(11) (∗) β −1 ∑ (∗)(∗)(∗) βreg
reg


ζω,t
( k fω,k gk,t − ζω,t ) + ζω,t /βreg





(βreg < 1)
, (12)
Rβ(∗)reg = ∑(∗)(∗) βreg


/βreg

k γω,t,k ( fω,k gk,t /γω,t,k )(1 ≤ βreg )Q = ε1 log Φ + ε2 log Ψ,

(13)

(KL)
Φ = δω,t,l fω,kgk,t ,(KL)Ψ = γω,t,k hω,l ul,t .

(14)
(15)

The equality in (9) holds if and only if the auxiliary variables are set as follows:
∑ (∗)
(∗)
′
′
α(∗)
(16)
k,l,ω = ( fω,k hω,l )/( ω fω′,k hω ,l ),
∑ (∗)
(∗)
(∗)
γω,t,k = ( fω,k gk,t )/( k′ fω,k′ gk′,t ),(17)∑
δω,t,l = (hω,l ul,t )/( l′ hω,l′ ul′,t ),(18)
ε1 = Φ/(Φ + Ψ),(19)
ε2 = Ψ/(Φ + Ψ),
(20)
∑ (∗)
(∗)
(21)
ζω,t = k fω,k gk,t .
The update rules are obtained from the derivative of
the upper bound function (8) w.r.t. each objective variable and substitution of the equality condition (16)–(21),日本音響学会講演論文集

Spatial conditions of each dataset

1stθ = 45◦
θ = 45◦
θ = 45◦
θ = 45◦

Measure2nd
3rd
θ = 0◦
θ = 0◦◦
θ = 45
θ = 0◦
θ = 45◦
θ = 45◦
θ = 45◦
θ = 45◦4thθ = 0◦
θ = 0◦
θ = 0◦
θ = 45◦

as

∑
(EUC)

ω iω,t yω,t fω,kgk,t · ∑
(
)βreg

∑
(EUC) (EUC)(EUC) ∑
(EUC)sω,t +λ ω iω,t fω,kω iω,t fω,kk′ fω,k′ gk′,t(rt ≥ τ)∑
gk,t ← ,(KL) (KL) −1iω,t yω,t fω,k
sω,t
ω



gk,t · ∑
(
)βreg
∑

(KL)
(KL) ∑(KL)ω iω,t fω,k +λ ω iω,t fω,kk′ fω,k′ gk′,t(rt < τ)(22)∑iω,t yω,t ul,t Nω,thω,l ← hω,l · ∑ t,(23)
iul,t Dω,t + µPω,l
ω,t
t
∑

ω iω,t yω,t hω,l(rt ≥ τ)ul,t · ∑ω iω,t hω,l s(EUC)ω,t∑ul,t ← ,(24)(EUC) −1ul,t · ω iω,t∑yω,t hω,l sω,t(rt < τ)ω iω,t hω,lwhere Nω,t , Dω,t , and Pω,l are given by{1(rt ≥ τ)Nω,t = (KL) −1,sω,t(rt < τ){ (EUC)s(rt ≥ τ)Dω,t = ω,t,1(rt < τ)∑ (EUC) ∑ (EUC)′(rt ≥ τ)′ f ′
 k fω,kω ,k hω ,lPω,l = .
∑ (KL) ∑ ω (KL)
 k fω,k′,lfh(rt < τ)′′ωω ω ,k

(25)
(26)
(27)

In total, the update rules of superresolution-based SNMF
with frame-wise divergence diversity are defined as(22)–(24).

4 Experiments
4.1 Experimental Conditions
To confirm the eﬀectiveness of the proposed algorithm, we compared five methods, namely, penalized SNMF (PSNMF) [2] based on KL-divergence,
PSNMF based on EUC-distance, the conventional hybrid method using only EUC-distance, the conventional
hybrid method using only KL-divergence, and the proposed hybrid method that switches the divergence to the
optimal one framewisely. In this experiment, we usedstereo signals containing four melody parts (depicted inFig. 4) with three compositions (C1–C3) of instruments
shown in Table 1. These signals were artificially generated by a MIDI synthesizer, and the observed signals
Y were produced by mixing four sources with the same
power. The sources were mixed as Fig. 5, where thetarget source was always located in the center direction
with another interfering source. However, these stereo
signals were mixed down to a monaural format only
when we evaluate the separation accuracy of PSNMF
because PSNMF is a separation method for a monaural
input signal.

- 729 -

2014年3月

KL-divergence based PSNMF
EUC-distance based PSNMF
KL-divergence based hybrid method
EUC-distance based hybrid method
Proposed hybrid methodaaaaaaanaaaaaaaaaaaaaaaaaaaaaaagaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaa
angiag
¶ ED?E E E DE E E EE E EE E DD ED?E E E ED E E EE E EE E DD¶
¶ EE E EEE E E E E D EE E EEE E E E E D´ E EEEE E D E EEEE E D

Fig. 4 Scores of each part. The observed signal consists of four measures.
Left

2

12
10

6

2

4

0

3

1

8

4

Center
Right

SP1

SP2

SP3

SP4

Spatial patterns
Fig. 6 Average SDR scores of each method and eachspatial condition.

θFig. 5 Panning of four sources, where numbered black
circles represent locations of instruments.
We prepared four spatially diﬀerent dataset patterns of
the observed signals, SP1–SP4, as shown in Table 2. In
the hybrid method, many chasms were produced by directional clustering in the measures where θ = 45◦ compared with those of θ = 0◦ . Therefore, we can expect that
EUC-distance-based hybrid method is suitable for SP4
rather than the dataset of SP1.
In addition, we used the same MIDI sounds of the target instruments as supervision for a priori training. The
training sounds contained two octave notes that cover allnotes of the target signal in the observed signal. The
sampling frequency of all signals was 44.1 kHz. The
spectrograms were computed using a 92-ms-long rectangular window with a 46-ms overlap shift. The number of
iterations for training and separation were 500, and thethreshold value τ was set to 20%. Moreover, the numberof clusters used in directional clustering was 3, the number of a priori bases was 100, and the number of bases
for matrix H was 30. The weighting parameters λ and µwere empirically determined.
4.2 Experimental Results
We used the signal-to-distortion ratio (SDR) definedin [10] as the evaluation scores. SDR indicates the quality of the separated target sound, which includes the degree of separation between the target and other sounds
and the absence of artificial distortion.
Figure 6 shows the average SDR scores for eachmethod and each dataset pattern, where four instruments
are shuﬄed with 12 combinations in each of compositions C1–C3. Therefore, these results are the averages of
36 input signals. From this result, KL-divergence-based
hybrid method achieves high separation accuracy for thedataset of spatial patterns SP1, SP2, and SP3 because
these signals do not have much spectral chasms. On
the other hand, EUC-divergence-based hybrid method
achieves high separation accuracy for SP 4. This dataset
has many spectral chasms because the signals are always mixed with a wide panning (θ = 45◦ ), which yields
many chasms, and the extrapolation ability is highly required. In addition, the proposed hybrid method withframe-wise divergence diversity can always achieve better separation for any datasets regardless of the condition
whether many chasms exist or not. This is because theproposed method provides the appropriate diversity of

日本音響学会講演論文集

14

SDR [dB]anthe divergence and can automatically apply the optimal
divergence to each time frame.

5 Conclusion
In this paper, we propose a new divergence selection
method to separate the target signal using optimal divergence. The proposed method switches the optimal divergence in each time frame using a threshold value forthe rate of the chasms to separate and extrapolate the target signal with high accuracy. Experimental results showthat our proposed method can always achieve high separation performance under any spatial conditions.

References[1] D. D. Lee, H. S. Seung, “Algorithms for non-negative matrix
factorization,” Proc. Advances in Neural Information Processing Systems, vol.13, pp.556–562, 2001.
[2] D. Kitamura,
H. Saruwatari,
K. Yagi,
K. Shikano,Y. Takahashi, K. Kondo, “Robust Music Signal Separation Based on Supervised Nonnegative Matrix Factorization
with Prevention of Basis Sharing,” Proc. ISSPIT 2013, 2013.
[3] A. Ozerov, C. Fevotte, “Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation,” IEEE Trans. Audio, Speech and Language Processing,vol.18, no.3, pp.550–563, 2010.
[4] D. Kitamura,
H. Saruwatari,
K. Shikano,
K. Kondo,
Y. Takahashi, “Superresolution-based stereo signal separation via supervised nonnegative matrix factorization,” Proc.
DSP, no.T3C-2, 2013.
[5] D. Kitamura,
H. Saruwatari,
K. Shikano,
K. Kondo,
Y. Takahashi, H. Kameoka, “Divergence optimization basedon trade-oﬀ between separation and extrapolation abilities
in superresolution-based nonnegative matrix factorization,”
Proc. 2013 Autumn Meeting of ASJ, 1-1-6, pp.583–586,
2013.
[6] D. FitzGerald, M. Cranitch, E. Coyle, “On the use of the beta
divergence for musical source separation,” Proc. Irish Signals
and Systems Conference, 2009.
[7] S. Araki, H. Sawada, R. Mukai, S. Makino, “Underdetermined blind sparse source separation for arbitrarily arranged
multiple sensors,” Signal Processing, vol.87, no.8, pp.1833–
1847, 2007.
[8] D. Kitamura,
H. Saruwatari,
K. Shikano,
K. Kondo,
Y. Takahashi, “Superresolution-based stereo signal separation with regularization of supervised basis extrapolation,”
Proc. 3DSA 2013, no.S10-4, 2013.
[9] S. Eguchi, Y. Kano, “Robustifying maximum likefood estimation,” Technical Report of Institute of Statistical Mathematics, 2001.
[10] E. Vincent, R. Gribonval, C. Fevotte, “Performance measurement in blind audio source separation,” IEEE Trans. Audio,
Speech and Language Processing, vol.14, no.4, pp.1462–
1469, 2006.

- 730 -

2014年3月