INTEGRATION AND ADAPTATION OF HARMONIC AND INHARMONIC MODELS FORSEPARATING POLYPHONIC MUSICAL SIGNALS
Katsutoshi Itoyama,† Masataka Goto,‡ Kazunori Komatani,† Tetsuya Ogata,† Hiroshi G. Okuno†‡
Dept. of Intelligence Science and Technology
National Institute of Advanced Industrial
Graduate School of Informatics, Kyoto University
Science and Technology (AIST)
Sakyo-ku, Kyoto 606-8501 Japan
Tsukuba, Ibaraki 305-8568 Japan{itoyama, komatani, ogata, okuno} [at] kuis.kyoto-u.ac.jp
m.goto [at] aist.go.jp†

ABSTRACTThis paper describes a sound source separation method for polyphonic sound mixtures of music to build an instrument equalizer
for remixing multiple tracks separated from compact-disc recordings by changing the volume level of each track. Although such
mixtures usually include both harmonic and inharmonic sounds, thedif¿culties in dealing with both types of sounds together have notbeen addressed in most previous methods that have focused on eitherof the two types separately. We therefore developed an integrated
weighted-mixture model consisting of both harmonic-structure andinharmonic-structure tone models (generative models for the power
spectrogram). On the basis of the MAP estimation using the EM algorithm, we estimated all model parameters of this integrated model
under several original constraints for preventing over-training andmaintaining intra-instrument consistency. Using standard MIDI ¿lesas prior information of the model parameters, we applied this model
to compact-disc recordings and achieved the instrument equalizer.
Index Terms— Music, separation, equalizers, sound source separation, music understanding.
1. INTRODUCTION
Our goal is to build an instrument equalizer, INTER [1], that enables a user to remix multiple audio tracks corresponding to different
instruments in compact-disc (CD) recordings by changing the volume of each track. While most previous equalizers, such as graphic
equalizers and tone controls for bass and treble, change the volumeof each frequency band to adjust frequency characteristics, INTER
can change the volume of each instrument to adjust the mixing balance of instruments in CD recordings. Because it was dif¿cult to extract (i.e., demix) tracks from CD recordings, however, the proposed
INTER was partially achieved only for drums: a drum-sound equalizer, called INTER:D [1], that changes the volume of drum sounds(inharmonic sounds). We therefore aim to built fully functional INTER, i.e., to build an advanced instrument equalizer for more generalsound mixtures, including both harmonic and inharmonic sounds.
Our equalizer, for example, enables a user to boost the volume of theguitar part, cut that of the bass part, and apply an additional soundeffect to the saxophone melody part.
Our equalizer requires that the multiple audio tracks to be equalized can be separated from polyphonic sound mixtures, such as thoserecorded on CDs. Such mixtures consist of both harmonic sounds
from pitched instruments, such as the piano and Àute, and inharmonic sounds from unpitched instruments, such as drums. Even
pitched instruments generate inharmonic sounds around their attacks(right after their onset times). For example, the piano sounds have
inharmonic sounds when a hammer hits a piano string. A soundsource separation method for our equalizer must therefore be able todeal with both types of sounds.
Most previous sound-source-separation methods, however, individually dealt with either of the two types and had dif¿culties dealing with both types together. For example, sound source separation
methods for harmonic sounds have been reported [2, 3, 4, 5], andthose for inharmonic sounds, such as drum sounds, have also been

1­4244­0728­1/07/$20.00 ©2007 IEEEI ­ 57

reported [6, 7]. Goto [8] mentioned a theoretical way of integrating inharmonic sound models with harmonic sound models, but has
not evaluated it yet. Although there is another approach of using
the blind sound source separation without any assumptions on soundsources [9], it still has dif¿culty dealing with sound mixtures consisting of many sounds and has not been successfully applied to CD
recordings, including complex audio signals of musical pieces.
We therefore propose a sound source separation method using an
integrated weighted-mixture model that consists of both harmonicstructure and inharmonic-structure tone models (generative models
for the power spectrogram). These tone models are complementary because inharmonic-structure tone models can capture drumsounds and attacks of instrument sounds. The harmonic-structure
tone model is based on a parametric model that represents the harmonic components of a pitched sound (e.g., a guitar sound). It isrepresented by model parameters such as the overall amplitude, F0(fundamental frequency) trajectory, onset time, duration, timbre (relative amplitude of harmonic components), temporal power envelope
for each pitched sound. The inharmonic-structure tone model, on
the other hand, is based on a non-parametric model that directly represents the power spectrogram of an unpitched sound (e.g., a drumsound or the attack of a guitar sound) in the time-frequency domain.
Our method estimates all model parameters of this integrated
model for each musical note in the audio signal of a musical pieceon the basis of the MAP (Maximum A Posteriori Probability) estimation using the Expectation-Maximization algorithm. Because
the inharmonic-structure tone model is too Àexible, however, it canbe ¿tted to any sounds, and the input sound mixture is sometimes
only represented by the inharmonic-structure models. To solve thisover-training problem, we estimated the model parameters by using
their prior information and initial values that are given on the basisof multiple tracks in the standard MIDI ¿le (SMF) corresponding
to the target piece.1 Moreover, we used several original constraints
based on the Kullback-Leibler (KL) divergence. For example, if theinharmonic-structure tone model captures harmonic components ofa sound, it is penalized. In addition, to maintain the intra-instrument
consistency, we penalized the tone model of a sound if it is too different from that of other sounds from the same instrument. Because thepower spectrograms of both harmonic and inharmonic sounds werethus obtained, the instrument equalizer was achieved by regenerating
and remixing separated tracks.
2. PROBLEMS AND APPROACHES
Given the input audio signal of a musical piece and its standard
MIDI ¿le (SMF) is synchronized with the input signal (i.e., audiosynchronized transcription), the goal of our method is to separate theinput signal into multiple audio tracks corresponding to the MIDI
tracks of the SMF. Each MIDI track usually corresponds to a different musical instrument part. In other words, our method estimates
all parameters of the harmonic-structure and inharmonic-structure
1 We assume that the SMF has already been synchronized with the input
mixture by using audio-to-score alignment methods [10, 11, 12].

ICASSP 2007

tone models corresponding to all the notes in these separated tracks.
By playing back each track of the SMF on a MIDI sound module,
we prepared a sampled sound for each note. We call this a template sound and used this as prior information (and initial values) inthe estimation. Even if the SMF and template sounds are available,
we still have the following dif¿culties in separating complex soundmixtures.
1. The timbre of a template sound from the MIDI sound module isalways different from that of the corresponding actual sound in
the input signal because the different instrument is used.
2. Even if the same musical instrument is performed at the same F0
and duration, the produced sounds are different because of playing styles and expressions, such as vibrato and dynamics. Thesesounds still have consistency, though, compared to the sounds
produced from a different instrument.
We address these issues as follows.
1. Starting from the tone model initialized with a template sound,the method tries to minimize the timbral difference between thetone model and the spectrogram of the corresponding actual
sound through the estimation of the model parameters. This estimation can be considered the model adaptation. We use theintegrated weighted-mixture model for all notes of both pitched
and unpitched instruments.
2. The method tries to maintain the intra-instrument consistency
while allowing for differences in the tone models for individual
notes. This can be achieved by estimating the model parameters
under constraint by using the KL divergence between the model
parameters of each note and those averaged over all the notes ofthe same instrument (in the same MIDI track).
3. FORMULATION
The problem is to decompose the input power spectrogram,g (O) (c, f, t), into the power spectrogram corresponding to each musical note, where c is the channel (e.g., left and right), f is the frequency, and t is the time. Our method can theoretically deal withsound recordings of any number of sources/channels (even monaural). We assume there are K musical instruments in the g (O) , andeach instrument plays Lk musical notes. Here, the spectrogramof a template sound of l-th musical note performed by k-th mu(T )sical instrument is denoted by gk,l (f, t), and the model (the corresponding spectrogram) estimated for its musical note is denotedby hk,l (c, f, t). The template spectrogram is monaural because thesound localization information in SMFs is not reliable.
For the decomposition of g (O) (c, f, t) with the model,
hk,l (c, f, t), we introduce a distribution (decomposition) functionof the spectrogram, m(O) (k, l; c, f, t), which satis¿es
X (O)0 ≤ m(O) (k, l; c, f, t) ≤ 1 andm (k, l; c, f, t) = 1.
k,lThus, m(O) (k, l; c, f, t)g (O) (c, f, t) represents the separated spectrogram of the l-th note by the k-th instrument. To evaluate the ‘effectiveness’ of this separation, we use the KL divergence between
this m(O) g (O) and the estimated-model spectrogram, hk,l :X Z Z (O)J1 (k, l) =m (k, l; c, f, t)g (O) (c, f, t)cm(O) (k, l; c, f, t)g (O) (c, f, t)
df dt,
hk,l (c, f, t)
To evaluate the ‘effectiveness’ of the estimated model hk,l (c, f, t),(T )we use the KL divergence between gk,l and hk,l :
Z
Z
(T )
X
gk,l (f, t)(T )df dt.
J2 (k, l) =gk,l (f, t) log
hk,l (c, f, t)c
We use the following sum over k and l of the KL divergences as
the cost function of the ‘effectiveness’ of both the separation and themodels:X
J0 =(αJ1 + (1 − α)J2 ) ,
log

k,l

I ­ 58Musical Signal

MIDI File

Recording Process
Reference
Template

Integrated
Model
Separation Process

Model Adaptation
Separated
SpectrumFig. 1. Overview of separation and model adaptation.

where α (0 ≤ α ≤ 1) is a weight parameter. We can prevent theover-training of the models by gradually increasing α from 0 (i.e.,
the estimated model should ¿rst be close to the template spectrogram) through the iteration of the separation and adaptation (model
estimation).
The overall process is depicted in Figure 1. We minimize J0 by
iteratively applying the separation process of estimating m(O) andthe model adaptation of estimating hk,l one after the other; the othervariables are ¿xed during the iterations. Here, J0 is written as
X ZZ
J0 =
m(O) (k, l; c, f, t)g (O) (c, f, t)
c,k,l

log
+

X ZZ

m(O) (k, l; c, f, t)g (O) (c, f, t)
df dt
hk,l (c, f, t)
(T )

(T )

gk,l (f, t) log

c,k,l− λ(c, f, t)

X X ZZk,lgk,l (f, t)
df dt
hk,l (c, f, t)!

m(O) (k, l; c, f, t) df dt − 1 ,cwhere λ is a Lagrange undetermined multiplier. First, we minimize
J0 by optimizing the m(O) with the hk,l ¿xed. The partial derivatives of J0 are∂J0m(O) (k, l; c, f, t)g (O) (c, f, t)−λ= g (O) (c, f, t) log(O)
hk,l (c, f, t)
∂m
X
∂J0
and
=
m(O) (k, l; c, f, t) − 1.
∂λk,l

By solving the simultaneous equations:∂J0∂J0= 0 and
= 0,
∂λ
∂m(O)
we get the optimal m(O) (k, l; c, f, t):hk,l (c, f, t).
m(O) (k, l; c, f, t) = P
k,l hk,l (c, f, t)Second, we minimize J0 by optimizing the hk,l with the m(O) ¿xed.
Because this optimization depends on the model de¿nition of hk,l , itis described in Sections 4 and 5.
Note that the above optimization of m(O) and hk,l is equivalent
to the MAP estimation using the EM algorithm. This fact becomes4.2. Inharmonic-structure tone model

Table 1. Parameters of harmonic-structure tone model.

Symbolwk,lμk,l (t)vk,l,nuk,l,yτk,lY φk,lσk,lDescription
overall amplitude
F0 trajectory
relative amplitude of n-th harmonic component
coef¿cient of the temporal power envelope
onset time
duration (note that Y is constant)diffusion of a harmonic component along the frequency axis

clear if we introduce a Q function given byQ(θ, θ̃) =missing data pdf
observed pdf
}|
{z
}|
{
X ZZ zp(k, l|c, f, t, θ) g (O) (c, f, t) log p(k, l, c, f, t|θ̃) df dt
{z
}
|
c,k,l
+

X ZZ

c,k,l

complete data pdf}|{
zp(k, l, f, t) log p(k, l, c, f, t|θ̃) df dt,| {z }
prior pdf

p(k, l|c, f, t, θ) = P

p(k, l, c, f, t|θ)k,l p(k, l, c, f, t|θ).

From this Q function, we can obtain a new cost function, J:
X ZZ
Gk,l (c, f, t)
J=df dt,
Gk,l (c, f, t) log
hk,l (c, f, t)
c,k,l

where

Gk,l (c, f, t) = αm(O) (k, l; c, f, t)g (O) (c, f, t)(T )+ (1 − α)gk,l (f, t).
J and J0 are equivalent when they are minimized by optimizingm(O) and hk,l .
4. INTEGRATED MODEL
The model, hk,l (c, f, t), is the integrated weighted-mixture modelthat consists of both harmonic-structure and inharmonic-structure
tone models and is given by
hk,l (c, f, t) = rk,l,c (Hk,l (f, t) + Ik,l (f, t)) ,where Hk,l (f, t) denotes the harmonic-structure tone model for harmonic components, Ik,l (f, t) denotes the inharmonic-structure tonemodel for inharmonic components, and rk,l,c is their relative amplitude in each channel.
4.1. Harmonic-structure tone model
The harmonic-structure tone model is given by
−1
N Y
X
X
wk,l Fk,l,n (f, t)Ek,l,y (t),Hk,l (f, t) =n=1 y=0!
vk,l,n(f − nμk,l (t))2
Fk,l,n (f, t) = √
exp −
,
2
2σk,l
2πσk,land(t − τk,l − yφk,l )2
uk,l,yexp −Ek,l,y (f, t) = √2φ2k,l2πφk,lThe inharmonic-structure tone model is represented as a nonparametric power spectrogram. As we pointed out in Section 1, input
sound mixture is sometimes represented only by this very Àexible
inharmonic-structure models without any harmonic-structure models. To solve this problem, which is known as the problem of overtraining or over¿tting due to too many parameters, we introduce anoriginal constraint given by«
ZZ „I¯k,l (f, t)
I¯k,l (f, t) log− I¯k,l (f, t) + Ik,l (f, t) df dt.
βI2
Ik,l (f, t)
This constraint has the effect of making Ik,l (f, t) close to I¯k,l (f, t),which is obtained by smoothing Ik,l (f, t) with a Gaussian ¿lter
along the frequency axis.
4.3. Constraint of the intra-instrument consistency
The models hk,l (c, f, t) estimated for musical notes of the same
instrument should have similar but different parameter values as
discussed in Section 2. To maintain this intra-instrument consistency we introduce two constraints. The ¿rst constraint is for the
harmonic-structure „tone model:
«
X
v̄k,n
− v̄k,n + vk,l,n ,βvv̄k,n log
vk,l,nnwhere v̄k,n is an intra-instrument average of vk,l,n , which meansthat the relative amplitude of each harmonic component should be
similar for a particular instrument. The second constraint is for theinharmonic-structure
tone model:«
ZZ „I¯k (f, t)¯− I¯k (f, t) + Ik,l (f, t) df dt,
Ik (f, t) log
βI1
Ik,l (f, t)¯
where Ik (f, t) is an intra-instrument average of Ik,l (f, t), whichmeans that the power spectrogram itself should be similar for a particular instrument.
5. MODEL ADAPTATIONAs described in Section 3, we can now minimize J0 , i.e., J, by optimizing the hk,l with the m(O) P¿xed. We assume that J is decomposed into Jk,l , given as J =k,l Jk,l . Given that λr , λv , andλu are Lagrange multipliers for rk,l,c , vk,l,n , and uk,l,y , the update
equations for each variable of the model, hk,l , are derived by minimizing J, which consists of the following sub-cost function:
Jk,l =(H)
X ZZ
Gk,l,n,y (c, f, t)(H)
Gk,l,n,y (c, f, t) log
rk,l,c wk,l Fk,l,n (f, t)Ek,l,y (t)
c,n,y
!
(H)

− Gk,l,n,y (c, f, t) + rk,l,c wk,l Fk,l,n (f, t)Ek,l,y (t) df dt
ZZ(I)X
Gk,l (c, f, t)(I)+
Gk,l (c, f, t) log
rk,l,c Ik,l (f, t)
c
!

!
,

(I)

where the parameters of this model are listed in Table 1. This model
was designed by referring to the harmonic-temporal-structured clustering (HTC) source model [4].
In the original HTC model, μk,l (t) was de¿ned as a polynomial function, but here, we de¿ne μk,l (t) as a more Àexible nonparametric function. Because this function is not constrained, it
might cause temporal discontinuities in the estimated F0 trajectory
μk,l (t). To prevent these discontinuities, we introduce an original
constraint given byZZZμ̄k,l (t)βμ μ̄k,l (t) log
dt withμ̄k,l (t) dt = μk,l (t) dt.
μk,l (t)This constraint makes μk,l (t) close to μ̄k,l (t), which is obtained bysmoothing μk,l (t) with a Gaussian ¿lter along the time axis.

I ­ 59+ λr− Gk,l (c, f, t) + rk,l,c Ik,l (f, t) df dt
!
!
!
X
X
X
rk,l,c −1 + λv
vk,l,n −1 + λu
uk,l,y −1
c

n

y

«
X„
v̄k,n
− v̄k,n + vk,l,n+ βvv̄k,n log
vk,l,n«Zn„
μ̄k,l (t)− μ̄k,l (t) + μk,l (t) dt+ βμμ̄k,l (t) log
μk,l (t)«ZZ „I¯k (f, t)¯¯Ik (f, t) log+ βI1− Ik (f, t) + Ik,l (f, t) df dt
Ik,l (f, t)«
ZZ „¯
Ik,l (f, t)
I¯k,l (f, t) log− I¯k,l (f, t) + Ik,l (f, t) df dt.
+ βI2
Ik,l (f, t)

Table 3. Experimental results.

Table 2. Experimental conditions.
Frequency analysis
sampling rate
STFT window

44.1 kHz
2048 points with a Gaussian window
Parameters# of partials: N
20# of kernels in Ek,l,y (t): Y
10
0.1
βv
βμ
0.01
0.1
ςμ
βI1
0.1
0.1
βI2
ςI
100
MIDI sound modules
test data
YAMAHA MU2000
template sounds
Roland SD-90

No. 1
No. 2
No. 5
No. 7
No. 10
Average

Note that Z „«p(x)p(x) log− p(x) + q(x; θ) dx
q(x; θ)
andZ
Z
Z
p(x)
p(x) log
dx, withp(x) dx = q(x; θ) dx
q(x; θ)are equivalent when they are minimized about θ. The update equations are omitted because it would take a page to list them all.
6. EXPERIMENTAL RESULTS
As a preliminary experiment to evaluate our new approach and determine the effectiveness of the model adaptation, we tested our method
on a small database consisting of ¿ve musical pieces so that we could
measure the signal-to-noise ratio (SNR) between the original sounds
and the separated sounds.
6.1. Experimental conditions
Although our method is intended to deal with CD recordings, thisexperiment uses audio signals recorded from a MIDI sound module
for the following reasons.
1. Most music databases does not contain both SMFs synchronizedwith the musical pieces and master tracks, which are necessaryfor quantitative evaluation based on SNR, of the pieces with theseparated signals.
2. Our separation method cannot deal with singing voices because
representing singing voices by our models and generating templates of singing voices are dif¿cult.
The template sounds were recorded from a different MIDI soundmodule made by another manufacturer. We used ¿ve SMFs from the
RWC Music Database [13] (RWC-MDB-P-2001 Nos. 1, 2, 5, 8, and10). Because of the computational time and memory limitation, thelength of each musical signal was limited to 5 seconds. The detailsof the experimental conditions are listed in Table 2.
6.2. Experimental results
The results are listed in Table 3. Using the model adaptation, we
increased the SNR from 5.53 to 5.73 dB. We think that separated
audio signals with 8 dB SNRs are suf¿cient for some applications,such as our instrument equalizer. Musical piece No. 7 had the lowest
SNR because this piece included some tracks which had very few
musical notes. It was therefore dif¿cult to model such notes because
the constraints of the same musical instrument were inadequate. We
think it is possible to improve the SNR if we use a longer excerpt of
this particular musical piece.
Musical piece No. 1 had the highest SNR, but the SNR decreased after the model adaptation. Because this piece included
many piano and acoustic guitar sounds, we think this fall-off wascaused by the over-training of the models to those sounds. We haveto introduce a new constraint to prevent this over-training problem.
Audio demonstrations, including the original sound mixtures
and the sounds separated using our method, are available at the following URL. http://winnie.kuis.kyoto-u.ac.jp/˜itoyama/icassp2007/.

I ­ 60w/o adapt.
9.25
6.65
5.40
3.64
5.74
5.53

SNR [dB]
1st adapt.
9.15
6.89
5.45
3.91
6.21
5.70

3rd adapt.
9.11
6.97
5.47
3.99
6.30
5.73

7. CONCLUSION
We have described a sound source separation method based on an integrated weighted-mixture model that represents both harmonic andinharmonic sounds. We implemented this method and tested it on asmall databse to determine the effectiveness of our model adaption
by using the EM algorithm. On the basis of this method, we havealready implemented the instrument equalizer that enables a user
to remix multiple separated tracks by changing the volume level ofeach track. Although this integrated model is Àexible and powerful,
we have not evaluated it on music signals, including vocals and CD
recordings. We plan to apply our method to various sound sourcesand validate its effectiveness.
ACKNOWLEDGMENTS: This research was partially supported by the
Ministry of Education, Culture, Sports, Science and Technology (MEXT),
Grant-in-Aid for Scienti¿c Research and Informatics Research Center for
Development of Knowledge Society Infrastructure (COE program of MEXT,
Japan).

8. REFERENCES[1] K. Yoshii, M. Goto, and H. G. Okuno, “INTER:D: a drum sound equalizer for controlling volume and timbre of drums,” in Proc. EWIMT,2005, pp. 205–212.
[2] T. Virtanen and A. Klapuri, “Separation of harmonic sounds using
linear models for the overtone series,” in Proc. ICASSP, 2002, vol. 2,
pp. 1757–1760.
[3] M. Every and J. Szymanski, “A spectral-¿ltering approach to musicsignal separation,” in Proc. DAFx, 2004, pp. 197–200.
[4] H. Kameoka, T. Nishimoto, and S. Sagayama, “Harmonic-temporal
structured clustering via deterministic annealing EM algorithm for audio feature extraction,” in Proc. ISMIR, 2005, pp. 115–122.
[5] H. Viste and G. Evangelista, “A method for separation of overlapping
partials based on similarity of temporal envelopes in multichannel mixtures,” IEEE Transactions on Speech and Audio Processing, vol. 14,
no. 3, pp. 1051–1061, May 2006.
[6] M. Helen and T. Virtanen, “Separation of drums from polyphonicmusic using non-negative matrix factorization and support vector machine,” in Proc. EUSIPCO, 2005.
[7] D. Barry, D. Fitzgerald, E. Coyle, and B. Lawlor, “Drum source separation using percussive feature detection and spectral modulation,” inProc. ISSC, 2005, pp. 13–17.
[8] M. Goto, “A real-time music-scene-description system: PredominantF0 Estimation for detecting melody and bass lines in real-world audio
signals,” Speech Communication (ISCA Journal), vol. 43, no. 4, pp.
311–329, September 2004.
[9] M. Casey and A. Westner, “Separation of mixed audio sources by
independent subspace analysis,” in Proc. ICMC, 2000, pp. 154–161.
[10] P. Cano, A. Loscos, and J. Bonada, “Score-performance matching using HMMs,” in Proc. ICMC, 1999, pp. 441–444.
[11] N. Adams, D. Marquez, and G. Wake¿eld, “Iterative deepening for
melody alignment and retrieval,” in Proc. ISMIR, 2005, pp. 199–206.
[12] Arshia Cont, “Realtime audio to score alignment for polyphonic music instruments using sparce non-negative constraints and hierarchicalhmms,” in Proc. ICASSP, 2006, vol. II, pp. 641–644.
[13] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC music
database: Popular, classical, and jazz music databases,” in Proc. ISMIR, 2002, pp. 287–288.