IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

Our goal is to build an instrument equalizer (remixer) that enables a user
to change the volume of parts of polyphonic sound mixtures, such as stereo
compact-disc recordings that consist of several instruments. Given separate audio tracks corresponding to diﬀerent instrument parts, it is easy to build such anequalizer, but separating a mixture of these audio tracks into diﬀerent tracks isdiﬃcult. Although a number of sound source separation methods 1)–5) and automatic transcription methods 6)–9) have been studied, most of them still havediﬃculty dealing with music performed on both pitched instruments that have
harmonic sounds and drums that have inharmonic sounds. For example, mostseparation methods for harmonic sounds 2)–5) cannot separate inharmonic sounds,
while most separation methods for inharmonic sounds, such as drums 10) , cannot
separate harmonic ones. Although there are separation methods 1),7) that deal
with both harmonic and inharmonic sounds in theory, they have practical difﬁculties in separating complex polyphonic sound mixtures like those in popular
music because they need temporal and frequency sparseness of the input signals.
We therefore propose a sound source separation method 11) that represents the
input polyphonic audio signal as a mixture of both harmonic and inharmonic
tone models that correspond to musical notes. We assume that a standard MIDI
ﬁle (SMF) synchronized with the input audio signal is available as prior information 1 . By using an iterative algorithm for estimating the parameters of theharmonic and inharmonic models, the model parameters are initialized using template sounds recorded from a MIDI sound generator and are gradually improved
so that they represent sounds in the input mixture. Once the model parameters
are estimated, we can easily obtain the separated power spectrogram of eachmusical note. We found that the integration of harmonic and inharmonic models
increased separation performance. However, this method did not model timbre
varieties within each instrument since the template sounds are generated from a
MIDI sound generator with single performance style. The instrument body andthe performance style are usually diﬀerent from that of the input sound mixtures.
This diﬀerence decreases the separation performance.
We propose a new method that estimates the model parameters of the integrated model consisting of both harmonic and inharmonic models while considering timbre varieties caused by diﬀerent instrument bodies and various performance styles. First, the timbre varieties of each instrument are modeled as atimbre feature distribution by using various training sound samples of that in-†1 Kyoto University†2 National Institute of Advanced Industrial Science and Technology (AIST)

1 We can assume several musical service providers, such as online music distributers, havea huge database which contains both, and our method would useful with a business stylewhich the providers do separation and clients uses separated signals.

Regular Paper

Parameter Estimation for Harmonic and Inharmonic
Models by Using Timbre Feature Distributions
Katsutoshi Itoyama,†1 Masataka Goto,†2
Kazunori Komatani,†1 Tetsuya Ogata†1and Hiroshi G. Okuno†1
We describe an improved way of estimating parameters for an integrated
weighted-mixture model consisting of both harmonic and inharmonic tone models. Our ﬁnal goal is to build an instrument equalizer (music remixer) thatenables a user to change the volume of parts of polyphonic sound mixtures. To
realize the instrument equalizer, musical signals must be separated into eachmusical instrument part. We have developed a score-informed sound source separation method using the integrated model. A remaining but critical problem is
to ﬁnd a way to deal with timbre varieties caused by various performance styles
and instrument bodies because our method used template sounds to represent
their timbre. Template sounds are generated from a MIDI tone generator basedon an aligned score. Diﬀerence of instrument bodies between mixed signals
and template sounds causes timbre diﬀerence and decreases separation performance. To solve this problem, we train probabilistic distributions of timbrefeatures using various sounds to reduce template dependency. By adding a new
constraint of maximizing the likelihood of timbre features extracted from eachtone model, we can estimate model parameters that express the timbre moreaccurately. Experimental results show that separation performance improved
from 4.89 to 8.48 dB.

1. Introduction1757

c 2009 Information Processing Society of Japan


1758

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions

strument. Then the model parameters, i.e., separate sounds, are estimated sothat the timbre features extracted from each separated sound have the maximum
likelihood with its timbre feature distribution. Our method can thus be used to
improve source separation performance using the varieties of timbre.
2. Sound Source Separation Using Integrated Models
In this section, we deﬁne our sound source separation problem and the integrated model.
The sound source separation problem is to decompose the input power spectrogram, X(c, t, f ), into the power spectrogram corresponding to each musicalnote, where c, t, and f are the channel (e.g., left and right), the time, and thefrequency, respectively. We assume that X(c, t, f ) includes K musical instruments and the k-th instrument plays Lk musical notes. We use the tone model,
J(k, l, c, t, f ), to represent the power spectrogram of the l-th musical note fromthe k-th musical instrument ((k, l)-th note), and the power spectrogram of atemplate sound, Y (k, l, t, f ), to initialize the parameters of J(k, l, c, t, f ). Eachmusical note of the SMF is played back on a MIDI sound generator in advance torecord the corresponding template sound. Y (k, l, t, f ) is monaural because SMFs
may not include accurate sound localization (channel) information. Y (k, l, t, f )is normalized to satisfy the following relation:
 
 
X(c, t, f ) dt df = C
Y (k, l, t, f ) dt df,
(1)
c

k,l

where C is the total number of channels.
We approximate the power spectrogram is additive. This approximation isvalid when the sounds are harmonic and sparse. Note that the validity decreases
if many instruments play simultaneously.
For this source separation, we deﬁne this integrated model, J(k, l, c, t, f ), as
the sum of the harmonic-structure tone models, H(k, l, t, f ), and inharmonicstructure tone models, I(k, l, t, f ), multiplied by the whole amplitude of themodel, wJ (k, l), and the relative amplitude of each channel, r(k, l, c):J(k, l, c, t, f ) = wJ (k, l) r(k, l, c) H(k, l, t, f ) + I(k, l, t, f ) ,(2)
where wJ (k, l) and r(k, l, c) satisfy the following constraints:

IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

Table 1 Parameters of integrated model.
SymbolwJ (k, l)r(k, l, c)wH (k, l), wI (k, l)vH (k, l, m, n)τ (k, l)φH (k, l)ωH (k, l, t)
σH (k, l)vI (k, l, m, n)φI
ωI (n)σI (n, f )wJ (k, l) =k,l∀k, l :

Description
overall amplituderelative amplitude of each channel
relative amplitude of harmonic and inharmonic tone models
relative amplitude of n-th harmonic at time mφH (k, l)onset time
diﬀusion of a Gaussian distribution constructing power envelope of theharmonic tone model
F0 trajectory
diﬀusion of a harmonic component along the frequency axis
relative amplitude of n-th inharmonic frequency component at time
mφI
diﬀusion of a Gaussian distribution constructing power envelope of theinharmonic tone model
central frequency of the n-th inharmonic frequency component
diﬀusion of an inharmonic frequency component along the frequency
axis

1
C


X(c, t, f ) dt dfr(k, l, c) = C.

and

(3)
(4)

c

Our aim is to decompose the power spectrogram of each musical instrument soundinto the harmonic and non-harmonic components, like a sinusoidal modelling
decomposes an input signal into a sum of sinusoidals and residual parts.
All parameters of J(k, l, c, t, f ) are listed in Table 1. The harmonic model,
H(k, l, t, f ), is deﬁned as a constrained two-dimensional Gaussian mixture modeland is designed by referring to the harmonic-temporal-structured clustering(HTC) source model 12) (see Figs. 1 and 2). The inharmonic model, I(k, l, t, f ),has a similar structure to the harmonic model. The inharmonic tone model has
the same structure along the time axis as the harmonic tone model. Along thefrequency axis, the inharmonic tone model has a structure in which the Gaussian
kernels are located at equal intervals on the logarithmic frequency (see Fig. 3),
to prevent the inharmonic model depriving from the harmonic model of the harmonic component when these models have similar shapes. The deﬁnitions ofthese models are as follows:c 2009 Information Processing Society of Japan


1759

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions

Fig. 3 Frequency structure of inharmonic tone model.
Fig. 1 Temporal power envelope of harmonic tone model.

vI (k, l, m, n)I(k, l, m, n, t, f ) =2πφI (ωI (n + 1) − ωI (n − 1))(t − (τ (k, l) − mφI ))2(f − ωI (n))2· exp −
exp
−
,
2φ2I
2σI (n, f )2
where
ωI (n) = ωIa ((ωIb )n − 1),ωI (n) − ωI (n − 1)
σI (n, f ) =ωI (n + 1) − ωI (n)Fig. 2 Harmonic structure of harmonic tone model.

H(k, l, t, f ) = wH (k, l)

MH 
NH


H(k, l, m, n, t, f ),(5)

m=0 n=1vH (k, l, m, n)H(k, l, m, n, t, f ) =2πφH (k, l)σH (k, l)2(t − (τ (k, l) − mφH (k, l)))2(f − nωH (k, l, t))· exp −exp −,2φH (k, l)2
2σH (k, l)2
MI 
NI

I(k, l, t, f ) = wI (k, l)I(k, l, m, n, t, f ),m=0 n=1

IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

(6)

(8)

(9)
(f ≤ ωI (n)),(f > ωI (n))(10)

MH and NH are the number of Gaussian kernels representing the temporal powerenvelope and the harmonic components of the harmonic tone model, respectively,
and MI and NI are the number of Gaussian kernels of the inharmonic tonemodel representing the same as above. vH (k, l, m, n), vI (k, l, m, n), wH (k, l), andwI (k, l) satisfy the following conditions:
NH
MH 

∀k, l :vH (k, l, m, n) = 1,
(11)∀k, l :

m=0 n=1
MI NIvI (k, l, m, n) = 1,

and

(12)

m=0 n=1(7)(13)∀k, l : wH (k, l) + wI (k, l) = 1.
The goal of this separation is to decompose X(c, t, f ) into J(k, l, c, t, f ) by
estimating a spectrogram distribution function, ΔJ (k, l, c, t, f ), which satisﬁes
c 2009 Information Processing Society of Japan


1760

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions∀k, l, c, t, f : 0 ≤ ΔJ (k, l, c, t, f ) ≤ 1,
ΔJ (k, l, c, t, f ) = 1.
∀c, t, f :

and

(14)
(15)

k,l

With ΔJ (k, l, c, t, f ), the separated power spectrogram, XJ (k, l, c, t, f ), is obtained as(16)
XJ (k, l, c, t, f ) = ΔJ (k, l, c, t, f )X(c, t, f ).
Furthermore, let ΔH (k, l, m, n, t, f ) and ΔI (k, l, m, n, t, f ) be spectrogram distribution functions which decompose XJ (k, l, c, t, f ) into each Gaussian distributionof the harmonic and inharmonic models, respectively. These functions satisfy∀k, l, m, n, t, f : 0 ≤ ΔH (k, l, m, n, t, f ) ≤ 1,
(17)
∀k, l, m, n, t, f : 0 ≤ ΔI (k, l, m, n, t, f ) ≤ 1, and(18)
ΔH (k, l, m, n, t, f ) +ΔI (k, l, m, n, t, f ) = 1.
(19)∀k, l, t, f :m,nTo evaluate the ‘eﬀectiveness’ of this separation, we can use a cost function deﬁned as the Kullback-Leibler (KL) divergence from XJ (k, l, c, t, f ) toJ(k, l, c, t, f ): 
XJ (k, l, c, t, f )dt df.
(20)
XJ (k, l, c, t, f ) log
J(k, l, c, t, f )c
By minimizing the sum of the divergences over (k, l) pertaining to ΔJ (k, l, c, t, f ),we obtain the spectrogram distribution function and model parameters (i.e., themost ‘eﬀective’ decomposition).
By minimizing the divergence pertaining to each parameter of the integrated
model, we obtain model parameters estimated from the distributed spectrogram.
This parameter estimation is equivalent to a maximum likelihood estimation.
3. Timbre Varieties Representation Using Prior Distribution
In this section, we describe timbre varieties and timbre feature distributions
for estimating parameters of the model.
3.1 Timbre Varieties within Each Instrument
Even within the same instrument, diﬀerent instrument bodies have diﬀerent
timbres, although its timbral diﬀerence is smaller than the diﬀerence among
diﬀerent musical instruments. Moreover, in live performances, each musical note

IPSJ Journal

Vol. 50

Fig. 4 Overview of iterating the separation and parameter estimation.

m,n

No. 7

1757–1767 (July 2009)

could have slightly diﬀerent timbre according to the performance styles. Insteadof preparing a set of many template sounds to represent such timbre varieties
within each instrument, we represent them by using a probabilistic distribution.
We use parameters of the integrated model, (wH (k, l), wI (k, l)), vH (k, l, m, n),vI (k, l, m, n), to represent the timbre variety of instrument k by training Dirichlet
distributions, which are known as the conjugate priors of these weight parameters.
We deﬁned three distributions for each instrument:( 1 ) p(wH (k, l), wI (k, l)),( 2 ) p(vH (k, l, 0, 1), . . . , vH (k, l, MH − 1, NH )) and( 3 ) p(vI (k, l, 0, 1), . . . , vI (k, l, MI − 1, NI )).
The model parameters for training the prior distributions were extracted fromthe “RWC Music Database: Musical Instrument Sound” 13) (i.e., the parameters are estimated without any prior distributions). The probability distributionfunctions of these prior distributions are described as follows:p(wH (k, l), wI (k, l)) ∝ wH (k, l)αwH (k)−1 wI (k, l)αwI (k)−1 ,(21)vH (k, l, m, n)αvH (k,m,n)−1p(vH (k, l, 0, 1), . . . , vH (k, l, MH − 1, NH )) ∝
m,n

(22)

c 2009 Information Processing Society of Japan


1761

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature DistributionsvI (k, l, m, n)αvI (k,m,n)−1 ,p(vI (k, l, 0, 1), . . . , vI (k, l, MI − 1, NI )) ∝
m,n

(23)
where {αwH (k), αwI (k)}, {αvH (k, m, n)} and {αvI (k, m, n)} are the parametersof the prior distributions. We assume that the values of these parameters are
more than 1.
Let XH (k, l, m, n, c, t, f ) and XI (k, l, m, n, c, t, f ) be the decomposed power:
XH (k, l, m, n, c, t, f ) = ΔH (k, l, m, n, t, f )XJ (k, l, c, t, f ) and(24)
XI (k, l, m, n, c, t, f ) = ΔI (k, l, m, n, t, f )XJ (k, l, c, t, f ).
(25)
By minimizing the cost function, 
XH (k, l, m, n, c, t, f )
Q=
c,m,n

Fig. 5 Minimizing the additional costs.

XH (k, l, m, n, c, t, f )
· log
dt df
wJ (k, l)r(k, l, c)wH (k, l)H(k, l, m, n, t, f )+
XI (k, l, m, n, c, t, f )c,m,n

XI (k, l, m, n, c, t, f )dt df
wJ (k, l)r(k, l, c)wI (k, l)I(k, l, m, n, t, f )− (αwH (k) − 1) log wH (k, l) − (αwI (k) − 1) log wI (k, l)(αvH (k, m, n) − 1) log vH (k, l, m, n)
−

· log

m,n

−



(αvI (k, m, n) − 1) log vI (k, l, m, n) ,

(26)

m,n

where the latter three terms are additional costs by using the prior distribution,
we obtain the parameters by taking into account the timbre varieties as shownin Fig. 5. This parameter estimation is equivalent to a maximum A Posteriori
estimation. The parameter update equations are listed in the Appendix.
3.2 Previous Cost Function without Considering Timbre Feature
Distributions
For comparison with our previous study 11) , we also tested the previous costfunction 11) in which we used template sounds instead of timbre feature distributions to evaluate the ‘goodness’ of the feature vector. Let YH (k, l, m, n, t, f ) and

IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

YI (k, l, m, n, t, f ) be the decomposed template power:
YH (k, l, m, n, t, f ) = ΔH (k, l, m, n, t, f )Y (k, l, t, f ) and(27)
YI (k, l, m, n, t, f ) = ΔI (k, l, m, n, t, f )Y (k, l, t, f ).
(28)
The cost function, used in the previous study, can be obtained by replacing the negative log-likelihood (the terms about log wH (k, l), log wI (k, l),log vH (k, l, m, n), and log vI (k, l, m, n) in Eq. (26)) with the KL divergence fromthe power spectrogram of a template sound which is weighted by the relative
amplitude of each channel, r(k, l, c)Y (k, l, t, f ), to J(k, l, c, t, f ): Q=
XH (k, l, m, n, c, t, f )c,m,n

XH (k, l, m, n, c, t, f )dt df
wJ (k, l)r(k, l, c)wH (k, l)H(k, l, m, n, t, f )+
XI (k, l, m, n, c, t, f )

· log

c,m,n

XI (k, l, m, n, c, t, f )dt df
wJ (k, l)r(k, l, c)wI (k, l)I(k, l, m, n, t, f )+r(k, l, c)YH (k, l, m, n, t, f )

· log

c,m,n

c 2009 Information Processing Society of Japan


1762

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions
Table 2 List of SMFs excerpted from RWC Music Database. Instruments are abbreviated,
and are explained in Table 3.

r(k, l, c)YH (k, l, m, n, t, f )dt df
wJ (k, l)r(k, l, c)wH (k, l)H(k, l, m, n, t, f )+r(k, l, c)YI (k, l, m, n, t, f )

· log

c,m,n

r(k, l, c)YI (k, l, m, n, t, f )dt df .
· log
wJ (k, l)r(k, l, c)wI (k, l)I(k, l, m, n, t, f )(29)

4. Experimental Evaluation
We conducted experiments to conﬁrm whether the performance of the source
separation using the prior distribution is better than the one using the templatesounds. We separated sound mixtures which were generated by mixing musicalinstrument sounds in the “RWC Music Database: Musical Instrument Sound” 13)
according to the SMFs of the “RWC Music Database: Jazz Music” and “RWC
Music Database: Classical Music” 14) which were excerpted to be about 30 seconds. In this experiment, we compared the following two conditions:( 1 ) using the log-likelihood of timbre feature distributions (proposed method,
Section 3.1),( 2 ) using the template sounds (previous method 11) , Section 3.2).
4.1 Experimental Conditions
We used 20 SMFs in total, which are listed in Table 2: ten SMFs are classicalmusical pieces and the other ten SMFs are jazz pieces. We prepared musicalinstrument sounds of 15 instruments listed in Table 3 from the RWC Music
Database: Musical Instrument Sounds 13) with two performance styles and three
instrument bodies. We generated sound mixtures for the test (evaluation) data by
mixing the instrument sounds corresponding to the notes in the SMFs. Since weused two performance-style sets and three instrument bodies, six sound mixtures
were generated from a SMF.
The prior distributions were trained by using the rest of the instrument sounds.
We assumed that vH (k, l, m, n) and vI (k, l, m, n) can be decomposed as follows:
vH (k, l, m, n) = vH (k, l, m)vH (k, l, n) andvI (k, l, m, n) = vI (k, l, m)vI (k, l, n),
and we used prior distributions, p(vH (k, l, m)), p(vH (k, l, n)), p(vI (k, l, m)) andp(vI (k, l, n)), instead of p(vH (k, l, m, n)) and p(vI (k, l, m, n)).

IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

Data Symbol

Instruments

Classical No.2
Classical No.3
Classical No.12
Classical No.16
Classical No.17
Classical No.22
Classical No.30
Classical No.34
Classical No.39
Classical No.40
Jazz No.1
Jazz No.5
Jazz No.8
Jazz No.9
Jazz No.16
Jazz No.17
Jazz No.23
Jazz No.24
Jazz No.27
Jazz No.28

VN, VL, VC, CB, TR, OB, FG, FL
VN, VL, VC, CB, TR, OB, FG, CL, FL
VN, VL, VC, CB, FL
VN, VL, VC, CL
VN, VL, VC, CL
PF
PFPF
PF, VNPF, VNPFPF
EGEG
PF, EBPF, EBPF, EB, TS
PF, EB, TS
PF, AG, EB, AS, TS, BS
PF, AG, EB, AS, TS, BS

Ave. # ofsources
6.23
6.51
4.23
3.30
3.76
4.33
4.94
5.96
5.92
7.54
2.75
6.92
6.47
3.23
3.55
5.19
3.64
6.28
11.71
5.46

The experimental procedure was as follows:( 1 ) initialize the integrated model of each musical note using the corresponding
template sound,( 2 ) estimate all the model parameters from the input sound mixture, and( 3 ) calculate the signal-to-noise ratio (SNR) for the evaluation.
SNR is deﬁned as follow:(J)Xkl (c, t)2
1
SNR =
dt,
10 log10(J)
C(T1 − T0 ) c(Xkl (c, t) − Zkl (c, t))2
where


(J)
Xkl (c, t) = XJ (k, l, c, t, f ) dfandZkl (c, t) = Z(k, l, c, t, f ) df, (30)

T0 and T1 are the beginning and ending times of the input power spectrogram,X(c, t, f ), F0 and F1 are the beginning and ending frequencies, and Z(k, l, c, t, f )is the ground-truth power spectrogram corresponding to the (k, l)-th note (i.e.,
c 2009 Information Processing Society of Japan


1763

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions

Table 3 List of musical instruments. The instrument ID means the unique instrument
number in the RWC Music Database: Musical Instrument Sounds 13) .
Inst. name (Abbr.)

Inst. ID

Pianoforte (PF)Electric Guitar (EG)Electric Bass (EB)
Violin (VN)
Viola (VL)
Cello (VC)
Contrabass (CB)
Trumpet (TR)Alto Sax (AS)
Tenor Sax (TS)Baritone Sax (BS)Oboe (OB)Fagotto (FG)Clarinet (CL)
Flute (FL)

No.1
No.13
No.14
No.15
No.16
No.17
No.18
No.21
No.26
No.27
No.28
No.29
No.30
No.31
No.33

Perf. style
set A (Abbr.)
Normal (NO)Legato/Pick (LP)
Normal/Pick (PN)
Normal (NO)Normal (NO)Normal (NO)Normal (NO)Normal (NO)Normal (NO)Normal (NO)Normal (NO)Normal (NO)Normal (NO)Normal (NO)Normal (NO)

Perf. style
set B (Abbr.)
Staccato (ST)Vibrato/Pick (VP)
Normal/Two-ﬁnger (TN)Non-vibrato (NV)Non-vibrato (NV)Non-vibrato (NV)Non-vibrato (NV)Vibrato (VI)
Vibrato (VI)
Vibrato (VI)
Vibrato (VI)
Vibrato (VI)
Vibrato (VI)
Vibrato (VI)
Vibrato (VI)
Fig. 6 SNRs of separated signals. [dB]

Table 4 Experimental conditions.
Frequency
Analysis

Sampling rate
Analyzing method
STFT window
STFT shift
ConstantCMH
Parameters
NH
MI
NI
φI
ωIa
ωIb
MIDI sound generator for template sounds* Short-time Fourier Transform

16 kHzSTFT*
2048 points Gaussian
160 points (10 ms)
1
20
30
20
30
0.05
440.0
1.135
Roland SD-90

the spectrogram of an actual sound before mixing). We have original, i.e., before
mixing, source signals. If we obtain ‘completely’ separated signals, the SNRs ofthese signals must be positive inﬁnity, or the SNRs will decrease as the separation performance becomes worse. Other experimental conditions are shown in
Table 4.

IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

4.2 Experimental Results
The average of SNRs of six sound mixtures for each musical piece is shownin Fig. 6, and Fig. 7 shows the SNRs for each musical instrument and performance style. The SNRs improved from 4.89 to 8.48 dB in average by using theprior distributions. This result shows the robustness and eﬀectiveness of ourmodel parameter estimation method under the timbre diﬀerence between musical instrument sounds consisting of input sound mixtures and template sounds.
Template sounds were generated from only one musical instrument body and performance style. These bodies and styles would be diﬀerent from the ones of the
input mixture signals and this diﬀerence decreased the separation performance.
The SNRs of pianoforte (PF) show a diﬀerence of more than 10 dB between
the normal (NO) and the staccato (ST) styles, although the diﬀerence of other
instruments between styles is at most 5 dB. Pianoforte sounds with the staccato
style have long silence period because the duration of these sounds is shorter thaneach note in the test data. Noises in the silence period decrease the SNR eventhough the noises added to the separated signal is little.
The SNRs of the electric bass (EB) with the pick/normal (PN) style, contrabassc 2009 Information Processing Society of Japan


1764

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions

Fig. 7 SNRs of separated signals for each musical instrument.

(CB) with both styles, and trumpet (TR) with vibrato (VI) style decreased, as
shown in Fig. 7. This decrease is considered to be caused by the following reasons:( 1 ) the prior distributions with inappropriate parameter values,( 2 ) the frequency resolution in low-frequency area.
In the future, reason (1) could be corrected by using an appropriate prior distribution, such as a mixture of the dirichlet distributions. This approach is eﬀective
in dealing with the timbre diﬀerence caused by performance styles. Reason (2)could be corrected by increasing the length of the Short-time Fourier Transform(STFT) window or using a nonlinear frequency analysis method, such as the
wavelet transform.
4.3 Discussion
As shown in Fig. 8, there was a correlation between the SNR and the average
number of notes for each musical piece. The Pearson product-moment correlationcoeﬃcient of these values is −0.59. The average number of notes indicates thediﬃculty in separating the signal, and the average number can be used to evaluate
the test data itself. Fig. 9 shows the correlation of the averaged SNR for eachframe of each musical note and the number of notes in the corresponding frame.
The SNR in the frames in which the number of sources was less than 6 was

IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

Fig. 8 Correlation between SNR and average number of notes for each musical piece.

Fig. 9 Correlation between averaged SNR for each frame of each musical note and thenumber of notes performed in the corresponding frame.

more than 10 dB, and the SNR in the frames in which the number of sources
was more than 9 was less than 5 dB. The validity of the additive approximationof the power spectrogram decreases as the number of sources increases, and thiscauses the separation performance decrease. These results mean the additivec 2009 Information Processing Society of Japan


1765

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions

approximation is not eﬀective when many instruments play simultaneously. To
improve the performance of the source separation in these frames with a large
number of sources, we will have to consider:• restoration of the distorted signals, and• decomposition of completely additive spectrogram (i.e., a complex spectrogram).
5. Conclusion
We described a new parameter estimation method for an integrated model byusing the timbre feature distributions. We conﬁrmed the following results:( 1 ) our method increased the separation performance for most instruments,( 2 ) in several musical instrument sounds which have very short duration or low
frequency components, the separation performance decreased, and( 3 ) the separation performance was aﬀected to the validity of the additive
approximation.
Our separation framework can be used as an instrument recognition methodby regarding the prior distribution as a recognizer. Therefore, we plan to apply
our method to the recognition problem by extending it to parallel processing ofseparation and recognition. Future work will also include the application of theseparated signals to various music listening interfaces.
Acknowledgments This research was partially supported by the Ministryof Education, Science, Sports and Culture, Grant-in-Aid for Scientiﬁc Researchof Priority Areas, Primordial Knowledge Model Core of Global COE programand CrestMuse Project.
References
1) Casey, M. and Westner, A.: Separation of Mixed Audio Sources by Independent
Subspace Analysis, Proc. ICMC, pp.154–161 (2000).
2) Virtanen, T. and Klapuri, A.: Separation of Harmonic Sounds Using Linear Models
for the Overtone Series, Proc. ICASSP, pp.1757–1760 (2002).
3) Every, M. and Szymanski, J.: A Spectral-ﬁltering Approach to Music Signal Separation, Proc. DAFx, pp.197–200 (2004).
4) Woodruﬀ, J., Pardo, B. and Dannenberg, R.: Remixing Stereo Music with Scoreinformed Source Separation, Proc. ISMIR, pp.314–319 (2006).

IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)5) Viste, H. and Evangelista, G.: A Method for Separation of Overlapping Partials
Based on Similarity of Temporal Envelopes in Multichannel Mixtures, IEEE Trans.
Audio, Speech and Lang. Process., Vol.14, No.3, pp.1051–1061 (2006).
6) Klapuri, A.: Multiple Fundamental Frequency Estimation based on Harmonicity
and Spectral Smoothness, IEEE Trans. Speech and Audio Process., Vol.11, No.6,
pp.804–816 (2003).
7) Smaragdis, P. and Brown, J.C.: Non-negative Matrix Factorization for PolyphonicMusic Transcription, Proc. WASPAA, pp.177–180 (2003).
8) Bertin, N., Badeau, R. and Richard, G.: Blind Signal Decompositions for Automatic Transcription of Polyphonic Music: NMF and K-SVD on the Benchmark,Proc. ICASSP, pp.65–68 (2007).
9) Ryynänen, M. and Klapuri, A.: Automatic Bass Line Transcription from Streaming
Polyphonic Audio, Proc. ICASSP, pp.1437–1440 (2007).
10) Barry, D., Fitzgerald, D., Coyle, E. and Lawlor, B.: Drum Source Separation Using Percussive Feature Detection and Spectral Modulation, Proc. ISSC, pp.13–17(2005).
11) Itoyama, K., Goto, M., Komatani, K., Ogata, T. and Okuno, H.: Integration and
Adaptation of Harmonic and Inharmonic Models for Separating Polyphonic MusicalSignals, Proc. ICASSP, pp.57–60 (2006).
12) Kameoka, H., Nishimoto, T. and Sagayama, S.: Harmonic-temporal Structured
Clustering via Deterministic Annealing EM Algorithm for Audio Feature Extraction, Proc. ISMIR, pp.115–122 (2005).
13) Goto, M., Hashiguchi, H., Nishimura, T. and Oka, R.: RWC Music Database:Music Genre Database and Musical Instrument Sound Database, Proc. ISMIR,
pp.229–230 (2003).
14) Goto, M., Hashiguchi, H., Nishimura, T. and Oka, R.: RWC Music Database:
Popular, Classical, and Jazz Music Databases, Proc. ISMIR, pp.287–288 (2002).

Appendix: Derivation of the Parameter Update Equation
In this appendix, we describe the update equations of each parameter derived
from the M-step of the EM algorithm. By diﬀerentiating the cost function foreach parameter, the update equations were derived as follows:
XJ (k, l),CC
XJ (k, l, c, t, f ) dt dfr(k, l, c) =,XJ (k, l)
XH (k, l) + (αwH (k) − 1),wH (k, l) =XJ (k, l) + (αwH (k) − 1) + (αwI (k) − 1)wJ (k, l) =

(31)
(32)
(33)

c 2009 Information Processing Society of Japan


1766

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions

XI (k, l) + (αwI (k) − 1),
(34)
XJ (k, l) + (αwH (k) − 1) + (αwI (k) − 1)
XH (k, l, m, n, c, t, f ) dt df +(αvH (k, m, n)−1), (35)vH (k, l, m, n) = c
XH (k, l)+ m,n (αvH (k, m, n)−1)wI (k, l) =(t − mφH (k, l))XH (k, l, m, n, c, t, f ) dt df

c,m,nτ (k, l) =ωH (k, l, t) =XI (k, l, m, n, c, t, f ) dt df + (αvI (k, m, n) − 1)
, (36)XI (k, l) + m,n (αvI (k, m, n) − 1)cvI (k, l, m, n) =c,m,n

XH (k, l)nf XH (k, l, m, n, c, t, f ) df

c,m,nn2 XH (k, l, m, n, c, t, f ) df

−aφH (k, l) +φH (k, l) =

(37)

,

(38)

aφH (k, l)2 + 4bφH (k, l)XH (k, l),
2XH (k, l)(39)(f − nωH (k, l, t))2 XH (k, l, m, n, c, t, f ) dt df

c,m,n

σH (k, l) =,XH (k, l)

where

 

XJ (k, l) =XJ (k, l, c, t, f ) dt df,, (40)

(41)

c

 

XH (k, l) =XH (k, l, m, n, c, t, f ) dt df,(42)

XI (k, l, m, n, c, t, f ) dt df,(43)

Katsutoshi Itoyama received the B.E. degree in 2006 and the
M.S. degree in Informatics in 2008 from Kyoto University, Japan.
He is currently a Ph.D. candidate in Informatics, Kyoto University.
He is supported by the JSPS Research Fellowships for Young Scientists (DC1). His research interests include musical sound source
separation, music listening interfaces, and music information retrieval. He recieved the 24th TAF Telecom Student Technology
Award. He is a member of the IPSJ and IEEE.
Masataka Goto received the D.E. degree from Waseda University, Japan, in 1998. He is currently a Leader of Media Interaction
Group, Information Technology Research Institute at the National
Institute of Advanced Industrial Science and Technology (AIST).
He serves concurrently as a Visiting Professor at the Institute of
Statistical Mathematics and an Associate Professor (Cooperative
Graduate School Program) at University of Tsukuba. He received
24 awards, including the Commendation for Science and Technology by the Minister of MEXT “Young Scientists’ Prize”, the DoCoMo Mobile Science Awards“Excellence Award in Fundamental Science”, and IPSJ Best Paper Award.

c,m,n

 

XI (k, l) =c,m,naφH (k, l) = m(t − τ (k, l))XH (k, l, m, n, c, t, f ) dt df

and

(44)

c,m,n

bφH (k, l) = (t − τ (k, l))2 XH (k, l, m, n, c, t, f ) dt df .

(45)

c,m,n

(Received May 12, 2008)(Accepted April 6, 2009)(Original version of this article can be found in the Journal of Information Processing Vol.17, pp.191–201.)IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

c 2009 Information Processing Society of Japan


1767

Parameter Estimation for Harmonic and Inharmonic Models by Using Timbre Feature Distributions

Kazunori Komatani received his B.E. degree in 1998, his
M.S. degree in Informatics in 2000, and his Ph.D. in 2002, allfrom Kyoto University. He is currently an Assistant Professorof the Graduate School of Informatics, Kyoto University, Japan.
From 2008 to 2009, he was a Visiting Scientist at Carnegie Mellon
University, Pittsburgh, PA, USA. He has received several awards
including the 2002 FIT Young Researcher Award and 2004 IPSJ
Yamashita SIG Research Award, both from the Information Processing Societyof Japan (IPSJ). His research interests center on spoken language processing,
especially on spoken dialogue systems. He is a member of the IPSJ, Instituteof Electronics, Information and Communication Engineers (IEICE), Association
for Natural Language Processing (NLP), Japanese Society for Artiﬁcial Intelligence (JSAI), Association for Computational Linguistics (ACL), and International Speech Communication Association (ISCA).

Tetsuya Ogata received the B.S., M.S. and D.E. degrees in Mechanical Engineering in 1993, 1995, and 2000, respectively, fromWaseda University. From 1999 to 2001, he was a Research Associate in Waseda University. From 2001 to 2003, he was a Research
Scientist in the Brain Science Institute, RIKEN. Since 2003, he has
been a Faculty Member in the Graduate School of Informatics,
Kyoto University, where he is currently an Associate Professor.
Since 2005, he has been a Visiting Associate Professor of the Humanoid Robotics
Institute of Waseda University. His research interests “interaction emergence systems” including human-robot vocal-sound interaction, dynamics of human-robot
mutual adaptation, and active sensing with robot systems. Dr. Ogata received
the 2000 JSME Outstanding Paper Medal from the Japan Society of MechanicalEngineers, and the Best Paper Award of IEA/AIE-2005. He is a member of the
IPSJ, JSAI, RSJ, HIS, SICE, and IEEE.
Hiroshi G. Okuno received the B.A. and Ph.D. degrees fromthe University of Tokyo, Japan, in 1972 and 1996, respectively.
He is currently a Professor of the Graduate School of Informatics,
Kyoto University, Japan. He received various awards including
the Best Paper Awards of JSAI. His research interests include
computational auditory scene analysis, robot audition and musicscene analysis.

IPSJ Journal

Vol. 50

No. 7

1757–1767 (July 2009)

c 2009 Information Processing Society of Japan