1-1-6
Divergence optimization based on trade-oﬀ between separation and extrapolation
abilities in superresolution-based nonnegative matrix factorization ∗☆ Daichi Kitamura, Hiroshi Saruwatari, Satoshi Nakamura(Nara Institute of Science and Technology)

1 Introduction
In recent years, music and acoustic signal separation based on nonnegative matrix factorization (NMF)[1] has been a very active area of signal processing research. NMF for acoustical signals decomposes an input spectrogram into the product of a spectral basis matrix and its activation matrix. In particular, supervised
NMF (SNMF) [2, 3], which includes a priori training
with some sample sounds of a target instrument, can extract the target signal to some extent. However, for thecase of a mixture consisting of many sources, the source
extraction performance is markedly degraded when only
single-channel observation is available.
Multichannel NMF, which is a natural extension for
multichannel signal, has been proposed as an unsupervised method [4]. However, such an unsupervised separation is a diﬃcult problem because the decomposition isunderspecified. Hence, these algorithms involve strong
dependence on initial values and lack robustness.
As another means for addressing multichannel signal separation, a hybrid method, which concatenates
superresolution-based SNMF after directional clustering, has been proposed by the authors [5]. This method
uses index information generated by binary masking of
directional clustering to regard the spectral chasms as
unseen observations, and finally reconstructs the target
source components via spectrogram extrapolation using the supervised bases. In this paper, we extend themethod [5] to be a more generalized form introducing
new parametric cost functions for both separation andregularization based on β-divergence criterion. Also, we
discuss the optimal divergence for this method.

2 Conventional Method
2.1 SNMF
The unsupervised NMF approaches have diﬃculty inclustering the decomposed spectral patterns into a specific target instrumental sound. To solve this problem,
SNMF has been proposed [2, 3]. In particular, a penalized SNMF (PSNMF), which imposes a penalty term toforce supervised bases and other bases to become uncorrelated with each other, achieves good performance [3].
SNMF consists of two processes, a priori training andobserved signal separation, as described below in detail.
2.1.1 Training Process of Supervision
In SNMF, a priori spectral patterns (bases) should betrained in advance as a basis dictionary. Hereafter, we
assume that we can obtain a specific solo-played instrumental sound, which is the target of the separation task.
The trained bases are constructed by simple NMF as
Ytarget ≃ F Q,

(1)swhere Ytarget (∈ RΩ×T
≥ 0 ) is an amplitude (or a power) spectrogram of the specific signal for training, F (∈ RΩ×K
≥ 0 ) is∗Kazunobu Kondo, Yu Takahashi(Yamaha Corporation)a nonnegative matrix that involves bases of the target sigs
nal as column vectors, and Q(∈ R≥K×T0 ) is a nonnegative
matrix that corresponds to the activation of each basis of
F . In addition, Ω is the number of frequency bins, T s
is the number of frames of the training signal, and K is
the number of bases. Therefore, the basis matrix F constructed by (1) is used as the supervision (dictionary) of
the target instrumental signal.
To construct the basis matrix F and the activation matrix Q, a cost function is given by()
JNMF = D Ytarget ∥F Q ,(2)where D (·∥·) is an arbitrary distance function.
2.1.2 Signal Separation Process
The following equation represents the decompositionof SNMF using the trained supervision matrix F :Y ≃ F G + HU ,(3)

is an observed spectrogram,where Y (∈ RΩ×T
≥0 )G(∈ R≥K×T0 ) is an activation matrix that corresponds
to F , H(∈ RΩ×L
≥ 0 ) represents the residual spectral patterns that cannot be expressed by F G, and U (∈ R≥L×T
0 ) isan activation matrix that corresponds to H. Moreover,
T is the number of frames of the observed signal and L
is the number of bases of H. In SNMF, the matrices
G, H, and U are optimized under the condition that
F is known in advance. Hence, ideally, F G represents
the target instrumental components and HU represents
other components diﬀerent from the target sounds after
the decomposition.
2.1.3 Problem of PSNMF
PSNMF can extract the target signal particularly in thecase of a small number of sources. However, for thecase of a mixture consisting of many sources, the source
extraction performance is markedly degraded because of
the existence of instruments with similar timbre.
2.2 Directional Clustering and Its Hybrid Methodwith Superresolution-Based SNMF
Decomposition methods employing directional (spatial) information for the multichannel signal have also
been proposed as unsupervised techniques [6]. Thesemethods quantize the direction via time-frequency binary masking. Such directional clustering works well,even in an underdetermined situation. However, there
is an inherent problem that the sources located in thesame direction cannot be separated only using the directional information. Furthermore, the extracted signal islikely to be distorted because the signal has many spectral chasms resulting of the binary masking procedure asshown in Fig. 1.
To solve this problem, a hybrid method that concatenates superresolution-based SNMF after the directional
clustering has been proposed [5]. This SNMF algorithm
explicitly utilizes index information determined by timefrequency binary masking in directional clustering. For“超解像型非負値行列因子分解における分離性能と外挿能力のトレードオフに基づく最適なダイバージェンスの検討,” by 北村大地, 猿渡洋, 中村哲 (奈良先端大), 近藤多伸, 高橋祐 (ヤマハ株式会社).

日本音響学会講演論文集- 583 -

2013年9月

Directional clustering

Frequency

Center component

L-ch
Superresolutionbased SNMF

R-ch
Superresolutionbased SNMF

ISTFT

ISTFT
MixingExtracted signal

Spectrum separated bydirectional clustering

Amplitude

Index ofcenterclusterThis generalized divergence is a family of cost functions parameterized by a single shape parameter β thattakes Itakura-Saito divergence (IS-divergence), generalized Kullback-Leibler divergence (KL-divergence), andEuclidean distance (EUC-distance) as special cases (β =0, 1 and 2, respectively). Using (5), we can define theseparation and regularization costs as follows: βNMFyω,t zβω,tNMF −1 ∑ zω,t
JNMF (θ| Y , I, F ) = ω,t iω,t 
−
, (6)


 βNMF
βNMF − 1 ∑(∑)
Jreg (G| I, F ) = ω,t iω,t k fω,k gk,t βreg /βreg ,(7)∑∑zω,t = k fω,k gk,t + l hω,l ul,t ,(8)

Amplitude

Input stereo signal
L-ch
R-ch
STFT

Frequency

Reconstructed spectrumby superresolution

Fig. 1Signal flow of proposed hybrid method.

example, if the target instrument is localized in the center cluster along with the interference, superresolutionbased SNMF is only applied to the existing center components using index information (see Fig. 1). Therefore,
the spectrogram of the target instrument is reconstructedusing more matched bases because spectral chasms aretreated as unseen, and these chasms have no impact on
the cost function in SNMF. In addition, the componentsof the target instrument lost after directional clustering
can be extrapolated using the supervised bases. In otherwords, the resolution of the target spectrogram is recovered with the superresolution by the supervised basis extrapolation. Furthermore, a regularization term isadded in the cost function to this SNMF for avoiding thebasis extrapolation error.

3 Superresolution-Based SNMF
Generalized Cost Function

with

3.1 Generalized Cost Function Using β-Divergence
In this section, we derive the update rules ofthe superresolution-based SNMF with generalized costfunctions. Here, the index matrix I(∈ RΩ×T
{0, 1} ) is obtained
from the binary masking preceding the directional clustering. This index matrix has specific entries of unity
or zero, which indicate whether or not each grid of thespectrogram belongs to the target directional cluster. The
generalized form of the cost function in superresolutionbased SNMF is defined using the index matrix as
J(θ) =JNMF (θ| Y , I, F ) + λJreg (G| I, F ) + µ∥F

T

H∥2F ,(4)

where θ = {G, H, U } is the set of objective variables,
λ and µ are the weighting parameters for each cost,and ∥ · ∥F represents the Frobenius norm. The firstterm JNMF represents the main cost for the separationof superresolution-based SNMF, the second term Jreg
represents the cost for the regularization, and the third
penalty term ∥F T H∥2F indicates that F and H are forced
to become uncorrelated with each other to avoid sharing
the same basis [3].
In this paper, we propose to use β-divergence as theseparation and regularization costs to generalize the criteria of these costs and find optimal divergences fit forsuperresolution-based SNMF. The β-divergence is defined asyβ
xβ yxβ−1+ −(β ∈ R\ {0, 1} ) β((β − 1) β ) β − 1
Dβ (y∥x) = 
y log y − log x + x − y (β → 1) . (5)





y
y


 − log − 1(β → 0)x
x

日本音響学会講演論文集

where yω,t , fω,k , gk,t , hω,l , and ul,t are the nonnegative
entries of matrices Y , F , G, H, and U , respectively,iω,t is the entry of index matrix I, which maps the values of one and zero onto the time-frequency (ω-t) region, · represents the binary complement of each entry
in the index matrix, and βNMF and βreg are the parameters that define the shape of divergences of separation
and regularization. Since the divergence JNMF is only
defined in grids whose index is one, SNMF treats only
the valid components except for the spectral chasms. In
addition, the regularization cost Jreg , which corresponds
to the grids of spectral chasms, forces to minimize thetarget component F G in proportion to the number ofzeros in index matrix I in each frame. Hence, the supervised bases are chosen so as to minimize F G to avoid
the extrapolation error.
3.2 Auxiliary Function and Update Rules
The update rules based on (4), (9), and (13) are obtained by the auxiliary function approach, similarly to[7]. First, we define the upper bound function for JNMF .
The first term of JNMF is convex for βNMF ≥ 1 and concave for βNMF < 1, and the second term is convex for
βNMF ≥ 2 and concave for βNMF < 2. Applying Jensen’s
inequality to the convex function and tangent line inequality to the concave function, we can define the upper′bound function JNMF
using auxiliary variables∑ αω,t,k ≥ 0,
γ
≥
0,
η
≥
0,
η
1
2 ≥ 0, and σω,t that satisfyk αω,t,k = 1,
∑ω,t,l
γ
=
1,
and
η
+η
=
1
as
ω,t,l
1
2
l
∑
(β
)
′
JNMF ≤ JNMF
= ω,t iω,t Pω,tNMF ,

(9)

where
 (βNMF )(βN−yω,t Mω,tNMF −1) (βNMF < 1)


 ω,t
(βNMF ) (βNMF )
(βPω,t = Mω,t −yω,t Mω,tNMF −1) (1 ≤ βNMF ≤ 2) , (10)M(βNMF ) −y N (βNMF −1) (βω,t ω,t
NMF > 2)
ω,t
{
∑
(
)βNMF
1
(β
)
Mω,tNMF =k αω,t,k η1 fω,k gk,t /αω,t,k η1
βNMF
}
∑
(
)
(11)
+ l γω,t,l η2 hω,l ul,t /γω,t,l η2 βNMF ,)(βNMF )
βNMF −1 (βNMF(12)
Nω,t =σω,tzω,t − σω,t + σω,t /βNMF .
Second, we define the upper bound function for Jreg .
This term is convex for βreg ≥ 1 and concave for βreg <1. Similarly to (9)-(12), we can define the upper bound′function Jreg
using auxiliary variables αω,t,k and ρω,t as- 584 -′
Jreg ≤ Jreg=∑

(βreg )
ω,t iω,t Sω,t ,(13)

2013年9月

Table 1
Dataset
No. 1
No. 2
No. 3

Fig. 2

Scores of each part.

where (β )regRω,t(βreg ) Sω,t =  (βreg )Qω,t(βreg < 1)
(1 ≤ βreg ),()(β ) ∑
Qω,treg = k αω,t,k fω,k gk,t /αω,t,k βreg /βreg ,)βreg(β ) β −1 (∑Rω,treg =ρω,treg
k fω,k gk,t − ρω,t + ρω,t /βreg .

(14)
(15)
(16)

Third, we define the upper bound function forT2∥F∑ H∥F using auxiliary variables δk,l,ω ≥ 0 that satisfyδω k,l,ω = 1 as
(
)
∑
2 2
∥F T H∥2F ≤ k,l,ω fω,khω,l /δk,l,ω .
(17)Finally, using (9), (13), and (17), we can define theˆ asupper bound function J ′(θ, θ)
(
)
∑
′
2 2
′
+ k,l,ω fω,k+ Jreghω,l /δk,l,ω , (18)J ′(θ, θ̂) = JNMF
where θ̂ is the set of auxiliary variables. The update rules
are obtained from the derivative of J ′ w.r.t. each objective variable and substitution of equality condition w.r.t.
each auxiliary variable as follows: ∑βNMF −2 φ(βNMF )ω iω,t yω,t fω,k rω,tgk,t ←gk,t  ∑,βNMF −1+ λvk,tω iω,t fω,k zω,t
 ∑βNMF −2 φ(βNMF )t iω,t yω,t ul,t rω,t,hω,l ← hω,l  ∑βNMF −1+ µwω,lt iω,t ul,t zω,t∑
βNMF −2 φ(βNMF ) ω iω,t yω,t hω,l rω,tul,t ← ul,t  ∑,βNMF −1 
i
h
z
ω,t
ω,l
ω
ω,t
where rω,t , vk,t , wω,l , and φ (β) are given by∑∑rω,t = k′ fω,k′ gk′,t + l′ hω,l′ ul′,t ,∑(∑)
vk,t = ω iω,t fω,k k′ fω,k′ gk′,t βreg ,∑∑wω,l = k fω,k ω′ fω′,k hω′,l ,
1/(2 − β)(β < 1)
1(1 ≤ β ≤ 2) .
φ(β) = 1/(β − 1)
(2 < β)

(19)

(20)

(21)

(22)
(23)
(24)
(25)

Melody 2
FluteViolinHorn

Midrange
Piano
Harpsichord
Piano

Bass
Trombone
FagottoCellosetting four divergences and regularizations, namely, β =0, 1, 2, and 3. We used the same divergence (βNMF ) in the
training and separation processes. In this experiment,
we used stereo signals containing four melody parts (depicted in Fig. 2) with three compositions of instruments
shown in Table 1. These signals were artificially generated by a MIDI synthesizer, and the observed signals
Y were produced by mixing four sources with the same
power. The target source is always located in the center
direction along with another interfering source, and there
are two sources in the left- and right-hand sides, which
are located at θ = ±15◦ based on a sine law. In addition,
we used the same MIDI sounds of the target instruments
as supervision for a priori training. The training sounds
contained two octave notes that cover all notes of thetarget signal in the observed signal. The spectrograms
were computed using a 92-ms-long rectangular window
with a 46-ms overlap shift. The number of iterations forthe training was 500 and that for the separation was 400.
Moreover, the number of clusters of used in directional
clustering was 3, the number of a priori bases was 100,and the number of bases for matrix H was 30. In thisexperiment, the weighting parameters λ and µ were experimentally determined.
4.1.2 Experimental Results and Discussion
We used the signal-to-distortion ratio (SDR), sourceto-interference ratio (SIR), and sources-to-artifacts ratio(SAR) defined in [8] as the evaluation scores. SDR indicates the quality of the separated target sound, SIR indicates the degree of separation between the target andother sounds, and SAR indicates the absence of artificial
distortion.
Figure 3 shows the average SDR, SIR, and SAR foreach divergence and each regularization, where four instruments are shuﬄed with 12 combinations in eachcomposition. Therefore, these results are the averagesof 36 input signal patterns. From the SDR in Fig. 3, we
can confirm that the EUC-distance-based cost function(βNMF ) is an optimal divergence for the hybrid methodwith superresolution-based SNMF. This is of great interest because EUC-distance-based NMF cannot achieve
the best performance for common music source separation, but instead KL-divergence is often used (as also
noted in many papers, e.g., [9]). In fact, SDR in Fig. 3
indicates that KL-divergence is the best divergence for
PSNMF. In addition, we can confirm that the regularization with KL-divergence is slightly better than theother divergences but the diﬀerence is not obvious.
In summary, we can assert as follows.
• The optimal divergence of separation (βNMF ) diﬀers
between simple SNMF and superresolution-based
SNMF, regardless of the type of regularization.

4 Experiments
4.1 Optimal Divergence and Regularization for
Superresolution-Based SNMF
4.1.1 Experimental Conditions
In this experiment, we compared some evaluation
scores with various βNMF and βreg to find the optimal
divergence and regularization fit for superresolutionbased SNMF. We compared the simple PSNMF [3] andthe hybrid method with superresolution-based SNMF by

日本音響学会講演論文集

Compositions of musical instruments

Melody 1
Oboe
TrumpetClarinet• We can speculate that this marked shift of optimal divergence in the SNMF methods is due tobalance of separation and extrapolation abilities;
this is because superresolution-based SNMF should
achieves both separation and extrapolation simultaneously, as described in Sect. 2.2.
The latter issue will be addressed in the next subsection.

- 585 -

2013年9月

0

PSNMF
Proposed hybrid method (βreg = 0)
Proposed hybrid method (βreg = 2)

12 (a)

Proposed hybrid method (βreg = 1)
Proposed hybrid method (βreg = 3)

10

20 (b)

8

8
6
4

15

SAR [dB]

SIR [dB]10

SDR [dB]

(c)

10
5

2
0

0

Fig. 3

1
2
Value of βNMF

0
0

1
2
Value of β NMF

3

0

1
2
Value of β NMF

3Average scores: (a) shows SDR, (b) shows SIR, and (c) shows SAR for each divergence and regularization.
Performance20
15

SAR [dB]

4
2

0

3

6

10

Total performance
Separation Extrapolation

5
Value ofFig. 500

1
2
3
Value of β NMF

Trade-oﬀ between separation and extrapolation abilities. Total performance gets highest when βNMF = 2.

4

Fig. 4

Average SAR for each divergence. This score showsthat extrapolation ability decreases when βNMF is close to zero.

4.2 Extrapolation Ability in Superresolution-Based
SNMF
4.2.1 Experimental Conditions
To quantify the extrapolation ability, we applied
superresolution-based SNMF to the binary-masked signals that contain only the target instrument. The binarymasks were made by directional clustering, which was
the same as the previous experiments. Therefore, theSAR of this experiment indicates the net extrapolation
ability. The parameter βNMF was set to 0, 1, 2, 3, and 4.
4.2.2 Experimental Results and Discussion
Figure 4 shows the average SAR of 36 input signal patterns for each divergence. From this result,superresolution-based SNMF loses the extrapolation
ability with decreasing βNMF . If βNMF is close to zero,
the spectral bases become more sparsity-aware, and thisproperty is suitable for normal NMF-based music source
separation. However, for superresolution-based SNMF,
which attempts to fit the trained bases using spectral
components except chasms, sparsity-aware bases are notsuitable because it is diﬃcult to extrapolate the target signal component using such sparse bases. Therefore, it can be confirmed that the optimal divergence forsuperresolution-based SNMF is shifted to EUC-distancerather than KL-divergence because of the trade-oﬀ between separation and extrapolation abilities, as illustrated in Fig. 5.

5 Conclusions
In this paper, we address a stereo signal separation
problem and derive the generalized multiple update rules
for superresolution-based SNMF based on β-divergence.
Also, we discuss about the optimal divergence for the

日本音響学会講演論文集

hybrid method based on the trade-oﬀ between separation and extrapolation abilities. From the experimental
results, it can be confirmed that EUC-distance is an optimal divergence for superresolution-based SNMF.

References[1] D. D. Lee, H. S. Seung, “Algorithms for non-negative
matrix factorization,” Neural Info. Process. Syst., vol.13,
pp.556–562, 2001.
[2] P. Smaragdis, B. Raj, M. Shashanka, “Supervised andsemi-supervised separation of sounds from singlechannel mixtures,” Proc. LVA/ICA 2010, LNCS 6365,
pp.140–148, 2010.
[3] K. Yagi, Y. Takahashi, H. Saruwatari, K. Shikano,
K. Kondo, “Music signal separation by orthogonality
and maximum-distance constrained nonnegative matrix
factorization with target signal information,” Proc. AES
45th, 2012.
[4] A. Ozerov, C. Fevotte, “Multichannel nonnegative matrix factorization in convolutive mixtures for audio
source separation,” IEEE Trans. Audio, Speech and Language Processing, vol.18, no.3, pp.550–563, 2010.
[5] D. Kitamura, H. Saruwatari, K. Shikano, K. Kondo,
Y. Takahashi, “Superresolution-based stereo signal separation via supervised nonnegative matrix factorization,”
Proc. DSP, no.T3C-2, 2013.
[6] S. Araki, H. Sawada, R. Mukai, S. Makino, “Underdetermined blind sparse source separation for arbitrarily
arranged multiple sensors,” Signal Processing, vol.87,
no.8, pp.1833–1847, 2007.
[7] M. Nakano, H. Kameoka, J. Le Roux, Y. Kitano,
N. Ono, S. Sagayama, “Convergence-guaranteed multiplicative algorithms for non-negative matrix factorization with beta-divergence,” Proc. MLSP, 2010.
[8] E. Vincent, R. Gribonval, C. Fevotte, “Performancemeasurement in blind audio source separation,” IEEE
Trans. Audio, Speech and Language Processing, vol.14,
no.4, pp.1462–1469, 2006.
[9] D. FitzGerald, M. Cranitch, E. Coyle, “On the use of thebeta divergence for musical source separation,” Proc.
Irish Signals and Systems Conference, 2009.

- 586 -

2013年9月