3-1-11
種々の実楽器信号に対する基底変形型教師あり NMF の分離精度評価∗☆北村大地, 猿渡洋, 鹿野清宏 (奈良先端大), 近藤多伸, 高橋祐 (ヤマハ株式会社)1

Introduction

In recent years, source separation based on nonnegative matrix factorization (NMF) [1], which is a
type of sparse representation algorithm, has been avery active area of signal processing research. NMF
for acoustical signals decomposes an input spectrogram into a product of a spectral basis matrix andits activation matrix. In particular, NMF has been aconvincing candidate used for source separation [2]in musical signal processing with a monaural format.
The methods of source separation based on NMF
are roughly classiﬁed into unsupervised and supervised algorithms. The former method attempts theseparation accompanied by various constraints, e.g.,
proposed in Refs. [2, 3]. However, these techniques
have a problem in spectral pattern clustering owing
to the blind approach. The latter method includes
a priori training, requiring some sample sounds ofa target instrument. In particular, a supervised
approach that introduces a penalized condition inoder to prevent some absences of the target soundachieves a good performance [4]. However, such supervised techniques have a critical problem that amismatch between the spectra trained in advanceand the target actual sound reduces the accuracy ofsource separation.
In this paper, we propose a new advanced supervised NMF algorithm that includes a deformableterm for the trained spectral bases and constraints
for making the bases to ﬁt into the real instrumental
sound. The experimental results show that the proposed method outperforms the conventional method[4] even with treating a mixture of real instruments.

s) is an amplitude spectrogram
where Ytarget (∈ RΩ×T
≥0
of a speciﬁc signal for training, F (∈ RΩ×K
≥ 0 ) is a nonnegative matrix that involves bases of the target sigs
nal as column vectors, and Q(∈ RK×T
) is a nonneg≥0
ative matrix that corresponds to the activation ofeach basis of F . In addition, Ω is the number of
frequency bins, Ts is the number of frames of the
training signal, and K is the number of bases.
In the decomposition of NMF, the cost function
can be constructed using some measures of the distance between Y and F Q as

JNMF = D (Y |F Q) ,where D (·|·) is an arbitrary distance function, e.g.,
Itakura-Saito divergence (IS divergence), and generalized Kullback Leibler divergence (KL divergence).
In this study, we propose to use KL divergence inthe cost function. The multiplicative update rules
for F and Q are given by∑∑−1
fω,k ts yω,ts qk,ts ( k′ fω,k′ qk′ ,ts )∑fω,k ←, (3)ts qk,ts∑∑−1
qk,ts ω yω,ts fω,k ( k′ fω,k′ qk′ ,ts )∑qk,ts ←, (4)
ω fω,kwhere yω,ts , fω,k , and qk,ts are the nonnegative entries of the matrices Y , F , and Q, respectively.
2.3 Signal separation process
The following equation represents the decomposition process of CSNMF using the trained supervision F :Y ≃ F G + HU ,2

Conventional method

Overview of constrained supervised
NMF
In the unsupervised approaches, it has diﬃcultly
in clustering the decomposed spectral patterns intoa speciﬁc target instrumental sound. Furthermore,each basis may be threatened to include a multiinstrumental spectral pattern. To solve this problem, constrained supervised NMF (CSNMF) has
been proposed as a supervised method [4]. CSNMF
consists of two processes, a priori training and observed signal separation, as described below in detail.
2.2 Training process of supervision
In CSNMF, as the supervision, a priori spectral
patterns (bases) should be trained in advance toachieve source separation. Hereafter, we assume
that we can obtain the instrumental sounds, whichis the target of the separation task. The trainedbases are constructed by NMF as∗(5)

where Y (∈ R≥ 0 ) is an observed spectrogram,G(∈ RK×T≥ 0 ) is an activation matrix that corresponds
to F , H(∈ RΩ×L
≥ 0 ) is the residual spectral patterns
that cannot be expressed by F G, and U (∈ RL×T
≥ 0 ) isan activation matrix that corresponds to H. Hence,F G represents the target instrumental components,and HU represents other diﬀerent components fromthe target sounds ideally. Moreover, L is the numberof frames of the observed spectrogram.
Ω×T

2.1

Ytarget ≃ F Q,

(2)

2.4

Cost function with constrained condition
After the decomposition by Eq. (5), F G is expected to represent the spectrogram that corresponds to the signal known in advance, and HU
expresses the spectrogram of other signals. However, if some of the spectral patterns in F and H are
the same, the corresponding activations separately
appear in G and U , degrading the separation performance. To cope with this problem, a constraint
is imposed in the cost functions of CSNMF as(1)

JCSNMF = D (Y |F G + HU ) + µ∥F T H∥F ,(6)“Evaluation of Separation Accuracy for Various Real Instruments Based on Supervised NMF with Basis
Deformation,” by Daichi Kitamura, Hiroshi Saruwatari, Kiyohiro Shikano (Nara Institute of Science andTechnology), Kazunobu Kondo, Yu Takahashi (Yamaha Corporation).

日本音響学会講演論文集- 1057 -2013年3月

where µ is weighting parameter, and ∥·∥F represents
Frobenius norm.
2.5 Multiplicative update rules of CSNMF
The update rules for Eq. (6) are given by∑∑∑−1
gk,t ω fω,k yω,t ( k′ fω,k′gk′ ,t + l′ hω,l′ul′ ,t)∑gk,t ←,
ω fω,k(7)∑∑∑−1hω,l t yω,t ul,t ( k′ fω,k′gk′ ,t + l′ hω,l′ul′ ,t)∑∑∑hω,l ←,′′ ′′t ul,t +2µ
k′ fω,k
ω ′ fω ,k hω ,l
(8)
∑
∑
∑
−1
ul,t ω hω,l yω,t ( k′ fω,k′gk′ ,t + l′ hω,l′ul′ ,t)∑ul,t ←,ω hω,l(9)where dω,k is the entry of the matrix D, which probably has positive and negative values, and η is theparameter that represents an allowable range of negative deformation of F .
3.2 Cost function with constraints
Since the deformation term D shares the activation G with F , D can be threatened to include
other instrumental components that simultaneously
sound with the target such as unison. This yields a
nonnegligible leakage of the undesired instrumental
components into the resultant output (F + D) G.
To avoid this phenomenon, the penalized terms fororthogonalization are imposed in the cost functionas
J =D(Y |(F +D)G+HU )+µ1 ∥F T D∥2FT+µ2 ∥F TH∥2F +µ3 ∥D TH∥2F +µ4 ∥(F+D) H∥2F ,(16)

where gk,t , hω,l , and ul,t are the nonnegative entriesof the matrices G, H, and U , respectively.
2.6 Problem of supervised method
The supervised techniques such as CSNMF involve an inherent problem that a mismatch between
the bases trained in advance and the target actual
sound reduces the accuracy of separation. Even if
the trained bases are constructed using the same
type of instrument as the target sound, the spectra
are diﬀerent according to, e.g., an individual style ofplaying and the timbre individuality for each instrument. Therefore, it is impossible to provide perfectsupervision and to predict more realistic supervisionin practice. To solve this problem, in the next section, we propose a new advanced CSNMF that includes a deformable term for the trained bases andthe additional constraint for making the bases to ﬁt
into the target sound.

where µ1 , µ2 , µ3 , and µ4 are the wighting parameters for each penalized term. These penalized terms
indicate that F , D, and H are forced to become
uncorrelated each other, and the target component
bases (F + D) and other component bases H also
become uncorrelated each other.

3

where rω,t , bω,k , and CJ are given by∑∑rω,t = k bω,k gk,t + l hω,l ul,t ,bω,k = fω,k + dω,k ,CJ = yω,t log yω,t − yω,t .

Proposed method

3.1

Supervised NMF with basis deformation
The proposed method also uses the pre-recorded
sound that is similar to the target instrument andavailable in advance for a priori training, and composes the trained bases F . For instance, when thetarget signal is the real speciﬁc instrumental sound,
it is allowed to use MIDI sound of the same typeof instrument because we can easily generate thetraining sound via MIDI. The decomposition modelis represented asY ≃ (F + D) G + HU ,(10)

where D(∈ RΩ×K
≥ 0 ) is a basis matrix which shares theactivation matrix G with F . In this decomposition,
to adapt the bases into the target sound that cannot be represented by F , another basis matrix D is
imposed as a deformation term for F . Furthermore,
D is constructed under the following constraints,
ηfω,k + dω,k ≥ 0,0 ≤ η ≤ 1,
gk,t ≥ 0,hω,l ≥ 0,
ul,t ≥ 0,

日本音響学会講演論文集

3.3 Multiplicative update rules
In this section, we derive the update rules basedon Eq. (16). Equation (16) can be rewritten as∑
J = ω,t (−yω,t log rω,t + rω,t + CJ )
∑
∑ ∑
∑
2
2
+µ1 k,k′ ( ω fω,k′ dω,k ) +µ2 k,l ( ω fω,k hω,l )
∑ ∑
∑ ∑
2
2
+µ3 k,l ( ω dω,k hω,l ) +µ4 k,l ( ω bω,k hω,l ) ,
(17)

(18)
(19)
(20)

Since it is diﬃcult to analytically derive the optimal
D, G, and H that minimize Eq. (17), we deﬁne an
auxiliary function, witch represents the upper boundof J as described bellow.
First, for the logarithmic term (hereinafter, referred to as Jlog ) in the ﬁrst term of the righthand side in Eq. (17), the upper bound function
Qlog is deﬁned using auxiliary variables∑ αk,ω,t ≥ 0,β∑l,ω,t ≥0, γ1 ≥0, and γ2 ≥0 that satisfy k αk,ω,t =1,l βl,ω,t =1, and γ1 +γ2 =1. Applying Jensen’s inequality to this, we have
Jlog =−yω,t log rω,t∑≤−yω,t k,l αk,ω,t βl,ω,t log (A + B) + CJlog∑≤−yω,t k,l αk,ω,t βl,ω,t (log Aγ1 +log B γ2 )+CJ′ log

(11)
(12)
(13)
(14)
(15)

- 1058 -≡ Qlog ,(21)

2013年3月

where A, B, CJlog , and CJ′ log are given by
A = βl,ω,t bω,k gk,t ,(22)
B = αk,ω,t hω,l ul,t ,(23)∑
CJlog = − k,l αk,ω,t βl,ω,t log αk,ω,t βl,ω,t ,(24)∑′
CJlog = CJlog− k,l αk,ω,t βl,ω,t (γ1 log γ1 +γ2 log γ2 ).
(25)
The equality in Eq. (21) holds if and only if the auxiliary variables are set to as follows:bω,k gk,tαk,ω,t = ∑
,
(26)
′
′
k′ bω,k gk ,thω,l ul,t,(27)
βl,ω,t = ∑′′l′ hω,l ul ,t
A
γ1 =
,
(28)
A+B
B
γ2 =
.
(29)
A+B
Second, for the penalized terms (hereinafter, referred to as Jp ) in Eq. (17), the upper bound function Qp is deﬁned using the auxiliary variables
δω,k′ ,k ≥∑0, ϵω,k,l ≥ 0, ∑
θω,k,l ≥ 0, and∑ λω,k,l ≥ 0 that′satisfyl θω,k,l =1, andl ϵω,k,l =1,
k δω,k ,k =1,∑
l λω,k,l=1. Similarly to Eq. (21), we obtain
∑ ∑
∑
∑
2
2
Jp =µ1 k′ ,k ( ω fω,k′ dω,k ) +µ2 k,l ( ω fω,k hω,l )
∑ ∑
∑ ∑
2
2
+µ3 k,l ( ω dω,k hω,l ) +µ4 k,l ( ω bω,k hω,l )
≤ µ1
+µ3

∑

k′ ,k,ω

2
2
2
fω,k
fω,k
h2ω,l
∑
′ dω,k+µ2 k,l,ωδω,k′ ,kϵω,k,l∑d2ω,k h2ω,lb2ω,k h2ω,l
∑
+µ
4
k,l,ω
k,l,ω
θω,k,lλω,k,l

≡Qp ,

(30)

where the equality in Eq. (30) holds if and only if
the auxiliary variables are set to as follows:fω,k′ dω,k,′ ′′
ω ′ fω ,k dω ,kfω,k hω,lϵω,k,l = ∑,′′ω ′ fω ,k hω ,ldω,k hω,lθω,k,l = ∑,ω ′ dω ′,k hω ′,lbω,k hω,lλω,k,l = ∑.
′′ω ′ bω ,k hω ,lδω,k′,k = ∑

(31)
(32)
(33)
(34)

Finally, by using Eq. (21) and Eq. (30), we candeﬁne the upper bound function J + for J as∑
J ≤ J + = ω,t (Qlog + rω,t + CJ ) + Qp .
(35)
The update rules for J + with respect to each variable are determined by setting the gradient to zero.
From ∂J + /∂dω,k = 0, we obtain∑()∑
yω,t αk,ω,t l βl,ω,t γ1
g
−
k,t
t
bω,k
2
∑ dω,k h2ω,l∑ fω,k′ dω,k
+2µ3 l+2µ1 k′δω,k′ ,k
θω,k,l∑ bω,k h2ω,l+ 2µ4 l= 0.
λω,k,l日本音響学会講演論文集(36)

By substituting Eqs. (26)-(29) and Eqs. (31)-(34)into Eq. (36), we can rewrite Eq. (36) as∑t gk,t +vω,k +2µ4∑

l hω,l∑ω ′ bω ′,k hω ′,l =where vω,k is given by∑∑vω,k =2µ1 k′fω,k′ ω′fω′,k′ dω′,k′∑∑+2µ3 l hω,l ω′ dω′,k hω′,l .

∑ yω,t gk,t
,
t
rω,t
(37)

(38)

Thanks to the constraints Eqs. (11)-(15), all theterms in Eq. (37) except vω,k are nonnegative. In
addition, vω,k can become both positive and negative values because dω,k has a possibility to be both
positive and negative. For keeping the nonnegativity of Eq. (37), we divide the update rules of dω,k incases that vω,k≥0 and vω,k<0. If vω,k≥0, both sidesof Eq. (37) are nonnegative. Then we can obtain the
update rule by multiplying both sides of Eq. (37) byηfω,k + dω,k , as∑−1
(ηfω,k + dω,k ) t yω,t gk,t rω,t∑∑dω,k ← ∑−ηfω,kt gk,t +vω,k +2µ4
l hω,lω ′ bω ′,k hω ′,l(vω,k ≥ 0) . (39)
If vω,k<0, we also have the update rule by transposing vω,k to the right-hand side and multiplying bothsides of Eq. (37) by ηfω,k + dω,k :(∑)−1(ηfω,k + dω,k ) t yω,t gk,t rω,t− vω,k∑∑∑−ηfω,kdω,k ←′′
t gk,t +2µ4
l hω,lω ′ bω ,k hω ,l(vω,k < 0) . (40)
Similarly to Eqs. (39) and (40), the update rules ofthe other variables are obtained as follows:∑−1hω,l t yω,t ul,t rω,t∑∑hω,l ← ∑ω ′ fω ′,k hω ′,l +wω,l +sω,lk fω,kt ul,t +2µ2(wω,k ≥ 0) , (41)
(∑
)
−1
hω,l
t yω,t ul,t rω,t − wω,l∑∑hω,l ← ∑′′t ul,t +2µ2
k fω,kω ′ fω ,k hω ,l +sω,l(wω,k < 0) , (42)∑−1
gk,t ω bω,k yω,t rω,t∑,(43)
gk,t ←ω bω,k∑−1
ul,t ω hω,l yω,t rω,t∑ul,t ←,(44)ω hω,lwhere wω,l and sω,l are given by∑∑wω,l = 2µ3 k dω,k ω′ dω′,k hω′,l ,∑∑sω,l = 2µ4 k bω,k ω′ bω′,k hω′,l .

4

(45)
(46)

Evaluation experiment

4.1 Experimental conditions
To conﬁrm the eﬀectiveness of the proposed algorithm, we compared the conventional method(CSNMF) and new NMF with Eqs. (39)-(44), applying them to the separation task for monaural
multiple instrumental sources. In this experiment,- 1059 -2013年3月

Table 1 Evaluation scores of conventional and proposed methods
TargetProposed method
Other
Conventional method
SDR
SIR
SAR
SDR
SIR
SAR
sound
sound
2.4
8.4
4.3
8.4
14.8
9.6
Piano
Clarinet
3.1
15.8
3.5
11.0
24.4
11.2
Piano
Trombone
0.1
1.8
7.3
0.7
2.6
7.2
Clarinet
Flute
3.2
14.6
3.7
9.6
23.9
9.8
Clarinet
Trombone
5.8
12.8
7.0
7.0
14.9
7.9
Piano
Flute
2.1
12.4
2.8
4.7
19.3
4.9
Trombone
Clarinet
we four types of real instruments, namely, piano,
clarinet, ﬂute, and trombone, as the target sound.
These sound sources were separately recorded, andthe observed signals Y were produced by mixing twosources selected from four sources with the input
SNR of 0 dB. In addition, we used artiﬁcial MIDI
sounds of the target instruments as supervision fora priori training. The training sounds contain two
octave notes that cover all notes of the target signal
in the observed signal. The sampling frequency of all
signals was 44.1 kHz. The spectrograms were computed using a 92-ms long rectangle window with a
46-ms overlap shift. The number of iterations for the
training was 500 and for separation was 400. Moreover, the number of a priori bases was 100, and thenumber of bases for the matrix H was 30. In this experiment, the wighting parameters were empiricallydetermined.
We used the signal-to-distortion ratio (SDR),
source-to-interference ratio (SIR), and sources-toartifacts ratio (SAR) deﬁned in Ref. [5] as the evaluation scores. SDR indicates the quality of separated
target sound. SIR indicates the degree of separation
between target and other sounds. SAR indicates absence of artiﬁcial distortion such as musical noise.
4.2 Experimental results
Table 1 shows the evaluated scores that are themaximum of SDR within 10 trials. Also, Fig. 1
shows an example of the separation result obtainedby the conventional and proposed algorithms. As
noted in Table 1, the separation performance is increased in the proposed method. The resultant
scores with respect to the mixture of clarinet andﬂute is slightly lower than those of other mixtures
because both of the spectral patterns are relatively
similar. Figure 1 shows that the target piano soundof the proposed method can be more enhanced thanthat of conventional CSNMF because the trainedbases are ﬁtted by the deformable ability in the proposed method.

5

Conclusions

In this paper, we propose a new advanced supervised NMF that includes the deformable term forthe trained spectral bases and constraints for making the bases to ﬁt into the target sound. Fromthe experimental results, it can be conﬁrmed thatthe proposed method increases the separation performance for the real instruments compared with theconventional method.

日本音響学会講演論文集

(a)

(b)

(c)

(d)

Fig. 1 Spectrograms of (a) observed signal consisting of piano and trombone, (b) oracle signal of thetarget piano signal, (c) extracted piano signal by
conventional method, and (d) extracted piano signal by proposed method.

References[1] D. D. Lee, H. S. Seung, “Algorithms for nonnegative matrix factorization,” Neural Info. Process. Syst., vol.13, pp.556–562, 2001.
[2] T. Virtanen, “Monaural sound source separation
by nonnegative matrix factorization with temporal continuity and sparseness criteria,” IEEE
Trans. Audio, Speech and Language Processing,vol.15, pp.1066–1074, 2007.
[3] H. Kameoka, M. Nakano, K. Ochiai, Y. Imoto,
K. Kashino, S. Sagayama, “Constrained andregularized variants of non-negative matrix
factorization incorporating music-speciﬁc constraints,” Proc. ICASSP, pp.5365–5368, 2012.
[4] K. Yagi,
Y. Takahashi,
H. Saruwatari,
K. Shikano, K. Kondo, “Music signal separation by orthogonality and maximum-distanceconstrained nonnegative matrix factorization
with target signal information,” Proc. Audio Engineering Society 45th International
Conference, 2012.
[5] E. Vincent, R. Gribonval, C. Fevotte. “Performance measurement in blind audio source separation,” IEEE Trans. Audio, Speech and Language Processing, vol.14, no.4, pp.1462–1469,
2006.

- 1060 -2013年3月