SIMULTANEOUS PROCESSING OF SOUND SOURCE SEPARATION AND MUSICALINSTRUMENT IDENTIFICATION USING BAYESIAN SPECTRAL MODELING
Katsutoshi Itoyama†∗ , Masataka Goto‡, Kazunori Komatani† , Tetsuya Ogata† , Hiroshi G. Okuno†† Graduate School of Informatics, Kyoto University, Japan∗ JSPS Research Fellowship for Young Scientists (DC1)‡ National Institute of Advanced Industrial Science and Technology (AIST), Japan
ABSTRACTThis paper presents a method of both separating audio mixtures
into sound sources and identifying the musical instruments of thesources. A statistical tone model of the power spectrogram, called
an integrated model, is defined and source separation and instrument identification are carried out on the basis of Bayesian inference.
Since, the parameter distributions of the integrated model depend oneach instrument, the instrument name is identified by selecting theone that has the maximum relative instrument weight. Experimental results showed correct instrument identification enables precise
source separation even when many overtones overlap.
Index Terms— Source separation, instrument identification,
Bayesian methods, spectrogram
1. INTRODUCTION
Musical instrument identification in complex musical audio mixturesand sound source separation of instrument sounds are challenging
problems in musical audio processing. These problems have thus
far been treated independently. For example, methods of musicalinstrument identification [1, 2] have been reported based on fundamental frequency (F0) estimates [3, 4, 5], tempo estimates and beattracking [6, 7, 8]. Methods of sound source separation have also
been reported for separating harmonic sounds [9, 10] and separating
percussive ones [11, 12]. Although methods of blind source separation and source (talker) identification have been reported [13] for
multi-channel audio signals recorded by using a microphone array,these methods cannot be applied to musical audio signals since mostmusical audio signals are monaural or stereo.
We believe that instrument identification and source separation
rely on each other, i.e., accurate instrument identification should help
source separation and high-quality source separation should simplify
instrument identification. This paper reports a method of both separating audio mixtures and identifying instruments for each sound.
The inputs are an audio mixture of instrument sounds, a number of
mixed sounds, and the rough onset time and F0 of each sound, andthe outputs are separated audio signals and the instrument name ofeach sound. We solved source separation as the decomposition of
the input power spectrogram based on the responsibility for each instrument sound, and instrument identification as the selection of thespectral tone model based on maximum A Posteriori approximation.
Since the distributions of the tone model parameters differ by instrument, we used prior distributions of the parameters, which weretrained by using a musical instrument sound database.
2. BAYESIAN SPECTRAL MODELINGIn this section, we define a stochastic tone model, which consists ofharmonic and inharmonic tone models, introduce prior distributions

978-1-4577-0539-7/11/$26.00 ©2011 IEEE3816

of the tone model parameters, and describe source separation and instrument identification methods based on Bayesian inference. Let anobserved power spectrogram, be a histogram on the two-dimensional
area of time and frequency, (t, f ). We assume that the spectrogram is obtained by using a short-time Fourier transform (STFT).
Since the elements of a power spectrogram are not generally integers, we approximate them as integers by multiplying a sufficiently
large number and rounding them. Let N be the number of samples
on the histogram, X = (x1 , . . . , xN ) be a whole set of samples, and
xn = (tn , fn ) (n = 1, . . . , N ) be each sample. We define separating sound sources as decomposing a power spectrogram of an audio
mixture into a power spectrogram that corresponds to each sound.
Let J be the number of musical instrument sounds performed in
the audio mixture and K be the number of candidate musical instruments. Onset time, duration, and pitch of each sound are given andthe instrument which performed the sound is unknown. Our goalis both of estimating instruments which performed each sound, i.e.,
instrument identification, and decomposing the input power spectrogram to each sound, i.e., source separation.
2.1. Harmonic and Inharmonic Tone Models
Let Yj (t, f ) be a spectral model which represents the power spectrogram of j-th instrument sound. Since the instrument which performed the sound is unknown, we define the power spectrogram
model of an instrument sound as the sum of K models with weightparameter bk|j :Yj (t, f ) =Kbk|j Yk|j (t, f ) .

(1)

k=1

We also represent the power spectrogram of the audio mixture by thesum of J models with weight parameter aj :JY(t, f ) =aj Yj (t, f ) .
(2)
j=1

To represent the power spectrogram of the k-th instrument of thej-th sound ((j, k)-th sound), we use a statistical tone model, called
an integrated harmonic and inharmonic models [14]. All musicalinstrument sound consists of harmonic sounds generated from a vibration of strings and air column, and inharmonic (percussive) ones
after musical instrument’s excitation. Power spectrograms of various musical instruments, e.g., clarinet and marimba which have large
harmonic and inharmonic energy, respectively, can be represented in
the same structure by using the integrated models.
The integrated model is defined as the sum of harmonic structure
model, YH|j,k (t, f ), and inharmonic structure model, YI|j,k (t, f ),with weight parameters cH|j,k and cI|j,k :Yk|j (t, f ) = cH|j,k YH|j,k (t, f ) + cI|j,k YI|j,k (t, f ) .

(3)

ICASSP 2011

The harmonic-structure tone model is defined as the constrained
two-dimensional Gaussian Mixture model (GMM), which is a product of two constrained one-dimensional GMMs. This model is designed by referring to the harmonic temporal structured clustering(HTC) source model [2]. The inharmonic-structure tone model isanalogously defined as a constrained two-dimensional GMM. The
temporal structures of these tone models are defined in similar form
to the harmonic one, but the frequency structures are defined as different forms. These models are defined as:
LH −1 MH

YH|j,k (t, f ) = dl,m|j,k Yl,m|j,k,H (t, f ) ,(4)

l=0 m=1

 
Yl,m|j,k,H (t, f ) = N t; τj + lρ, ρ2 N f ; mωj , σj2 ,YI|j,k (t, f ) =LI −1 MI 

(5)el,m|j,k Yl,m|j,k,I (t, f ) , and

(6)



Yl,m|j,k,I (t, f ) = N t; τj + l, 2 M(f ; m, λ, κ) ,

(7)





(x − μ)2
1
exp −
and
N x; μ, σ 2 = √
2σ 2
2πσ

(8)

l=0 m=1

where

M(x; μ, β, κ) =√
(β log(x/κ + 1) − μ)2
β
.
exp −
2
2π(x + κ)(9)

LH and MH are the number of Gaussian kernels that represent thetemporal and frequency parts of the harmonic model, and LI and MI
are the number of Gaussians that respectively represent those of theinharmonic model. The parameters are described as follows:• a = (a1 , . . . , aJ ). This parameter represents the relative weight(volume) of the j-th note. The a is in the standard (J − 1)simplex, i.e., a satisfies 0 ≤ aj ≤ 1 (j = 1, . . . , J) and
j aj = 1.
• bj = (b1|j , . . . , bK|j ). This parameter represents the relative
weight of the k-th instrument of the j-th sound. Each bj is in the(K − 1)-simplex.
• cj,k = (cH|j,k , cI|j,k ). This parameter represents the relative
weight of the harmonic and inharmonic components. Each cj,kis in the standard 1-simplex.
• dj,k = (d0,1|j,k , . . . , dLH −1,MH |j,k ). This parameter represents
the time-frequency energy distribution of the harmonic component. Each dj,k is in the standard (LH × MH − 1)-simplex.
• ej,k = (e0,1|j,k , . . . , eLI −1,MI |j,k ). This parameter represents
the time-frequency energy distribution of the inharmonic component. Each ej,k is in the standard (LI × MI − 1)-simplex.
• τj . This parameter represents the onset time of the j-th sound.
• ρ and . The LH ρ and LI  mean the duration of the harmoniccomponent and inharmonic component, respectively.
• ωj . This parameter represents the fundamental frequency.
• σj . This parameter represents the deviation in energy distribution
along the frequency axis.
• λ and κ. These parameters are coefficients that determine thearrangement of the Gaussian kernels for the frequency structureof the inharmonic model1 .
1 If λ and κ are set to 1127 and 700, respectively, λ log(f /κ + 1) isequivalent to the mel scale of f Hz.

3817

Since the integrated model is defined as a hierarchical weighted
mixture of Gaussian distributions, we introduce latent variables,
which specify the element distribution which generate sample xn :• z n = (z1n , . . . , zJn ). zjn = 1 means sample xn is generated fromthe j-th sound.
n
n
n
• zn
j = (z1|j , . . . , zK|j ). zk|j = 1 means sample xn is generated
from the (j, k)-th sound.
n
n
n
n
• zn
j,k = (zH|j,k , zI|j,k ). zH|j,k = 1 and zI|j,k = 1 means sample
xn is generated from the harmonic and inharmonic componentsof the (j, k)-th sound, respectively.
n
n
n
• zn
j,k,H = (z0,1|j,k,H , . . . , zLH −1,MH |j,k,H ). zl,m|j,k,H = 1
means sample xn is generated from the component whose temporary and frequency indices are l and m ((l, m)-th component)of the (j, k)-th sound’s harmonic component.
n
n
n
• zn
j,k,I = (z0,1|j,k,I , . . . , zLI −1,MI |j,k,I ). zl,m|j,k,I = 1 means
sample xn is generated from the (l, m)-th component of the(j, k)-th sound’s inharmonic component.
z n has a 1-of-J representation, i.e., one of zjn is 1 and the others
are 0, thus z n satisfies zjn ∈ {0, 1} and j zjn = 1. Other latentvariables have the same representation.
2.2. Prior distribution
We introduce prior distributions to prevent the model parametersfrom deviating in source separation and instrument identification.
For example, the energy distribution of the inharmonic component
generally converges just after sound excitation and decreases withtime, so usually el,m|j,k > el ,m|j,k (l < l ). Since acoustic features, e.g., relative amplitude of the harmonic components, are different for each musical instrument, we use different prior distributions of the model parameters for each instrument. Prior distributions are trained by estimating the model parameters for isolated musical instrument sounds from a sound database with noninformative
priors and averaging them. Let θ be a whole set of model parameters;
the prior distributions are described as:p(cj,k ) p(dj,k ) p(ej,k ) p(τj ) p(ωj , σj ) , (10)p(θ) = p(a)jp(a) = D(a; α)(α = (α1 , . . . , αJ )),(11)p(bj ) = D(bj ; β)(β = (β1 , . . . , βK )),(12)(γ j = (γj,H , γj,I )),(13)p(cj,k ) = D(cj,k ; γj )p(dj,k ) = D(dj,k ; δj )(δ j = (δj,0,1 , . . . δj,LH −1,MH )), (14)p(ej,k ) = D(ej,k ; εj )(15)

(εj = (εj,0,1 , . . . εj,LI −1,MI )),p(τj ) = N τj ; νk , ξk−1 , and p(ωj , σj ) = N ωj ; ϕk , σj−2 χk G σj−2 ; ηk , ζk .

(16)
(17)

Prior distributions are defined as a conjugate prior of the corresponding parameters. Parameters without prior distributions, ρ, , λ, andκ, are treated as constants. The D(·) and G(·) mean Dirichlet and
gamma distributions. The probabilistic density functions of these
distributions are given as follows except for normalizing factors:
D(x1 , . . . , xN ; φ1 , . . . , φN ) ∝Nxnφn −1 ,

(18)

n=1

G(x; η, ζ) ∝ xη−1 exp(−ζx) .

(19)

Let Z be a whole set of latent variables; a probabilistic modelwith the latent variables is described as:
N



n
n
n
p xn , z n , z np(X, Z, θ) =j , z j,k , z j,k,H , z j,k,I |θ p(θ) , (20)n=1The posterior probabilities in above equations are defined as: n p X, zjn = 1, Z−z n , θ dZ−zn dθ
p zj = 1|X =and (28)p(X, Z, θ) dZ dθp(bj |X) =



n
n
n
p xn , z n , z n
j , z j,k , z j,k,H , z j,k,I |θ =q(Z) =Nq(z n )

n=1

(21)
q(θ) = q(a)

j=1



n
n
n
n
p xn , z n
j , z j,k , z j,k,H , z j,k,I |zj = 1, θ =

J


q(z n
j)

j=1
J


q(bj )

j=1

K


z n

n
n
n
k|j
,
bk|j p xn , z n
j,k , z j,k,H , z j,k,I |zk|j = 1, θ

(22)

k=1



n
n
n
p xn , z n
j,k , z j,k,H , z j,k,I |zk|j = 1, θ =


z n
n
H|j,k
cH|j,k p xn , z n
×j,k,H |zH|j,k = 1, θnzI|j,knncI|j,k p xn , z j,k,I |zI|j,k = 1, θ
,


n
n
p xn , z j,k,H |zH|j,k = 1, θ =LH −1 MH 
zdl,m|j,k Yl,m|j,k,H (t, f ) l,m|j,k,H , and(29)

where Z−z n and θ−bj are the whole set of the latent variables andmodel parameters except for zn and bj , respectively, but it is impractical to directly calculate that integral since the parameter spaceis extremely huge. Therefore, we adopt a method of variational inference for estimating the posterior probability of the latent variables
and model parameters.
Let q(Z, θ) be a test distribution that approximates the posterior
distribution, p(Z, θ|X). We assume that the test distribution can befactorized as:q(Z, θ) = q(Z) q(θ),(30)

Fig. 1. Graphical model of integrated model.

J

zjn
 
n
n
n
n
,
aj p x n , z n
j , z j,k , z j,k,H , z j,k,I |zj = 1, θp(X, Z, θ) dZ dθ−bj,p(X, Z, θ) dZ dθ

(23)

(24)

K


n
n
q(z n
j,k ) q(z j,k,H ) q(z j,k,I ), and

k=1
K


(31)

q(cj,k ) q(dj,k ) q(ej,k ) q(τj ) q(ωj , σj )

k=1

(32)
An objective function for estimating the optimal q(z, θ) is defined as: p(X, z, θ)F[q] =dθ,(33)
q(z, θ) log
q(z, θ)z∈Z
where F[q] is a functional that depends on function q. The q
that maximizes F[q] most approximates the posterior distribution,p(Z, θ|X), under the factorization assumption.
To calculate an optimal test distribution that maximizes the objective function, we solved an Euler-Lagrange equation. All update
equations have been omitted since it would take a page to list them.

l=0 m=1

3. EXPERIMENTAL EVALUATIONnp xn , z n
j,k,I |zI|j,k = 1, θ =LI −1 MI  zel,m|j,k Yl,m|j,k,I (t, f ) l,m|j,k,I .

(25)

l=0 m=1

A graphical model of the observation is given in Fig. 1.
2.3. Bayesian inference
As we described, source separation is defined as the decompositionof the input power spectrogram. Decomposed power spectrogram of
the j-th sound, Xj (t, f ), is obtained by multiplying the responsibility, i.e., the expectation value of latent variable zjn , with the observed
power spectrogram:
p zjn = 1|X .
(26)
Xj (t, f ) =xn ∈{x|x∈X,x=(t,f )}Separated audio signals are obtained by an inverse STFT of the decomposed spectrograms.
Instrument identification is performed by model selection basedon Bayesian inference. The instruments are estimated by using
model selection based on maximum A Posteriori approximation:(Instrument of j-th note) = arg max bk|j p(b |X ) ,
k

j

(27)

where xf (x) means the expectation value of the random variable x
with the density function f (x), i.e., xf (x) = xf (x) dx .

We conducted an experiment to evaluate the efficiency of our source
separation and instrument identification methods. Given audio mixtures that consisted of two or three musical instrument sounds excerpted from the RWC Music Database: Musical Instrument Sound[15], the audio mixtures were separated into sources and instruments
were estimated. As shown in Table 1, eight musical instruments
were excerpted from the database and sounds were divided into subsets for 10-fold cross validation. The prior distribution of each instrument was created by averaging the model parameters estimatedfrom the training data (nine subsets). Audio mixtures were produced
from the combination of the instrument sounds for each data subset
except pairs consisting of the same instrument sounds. The constantparameters in the integrated models were set as listed in Table 2. The
performance of instrument identification and source separation wererespectively evaluated by using the accuracy rate and log spectral
distance defined as: T F 2 
  

20 log10 Xorg (t, f ) 
TF
(34)


X
sep (t, f )
t=0
f =0

Table 3 summarizes the accuracy rate of instrument identification and log spectral distance for the source instruments. The fagotto(FG), violin (VN), and clarinet (CL) have a high accuracy rate for
identification and short log spectral distances. This suggests that
correct instrument identification help to improve source separation.

3818

Table 1. Musical Instruments.
Inst. name (Abbr.)
Acoustic piano (PF)Violin (VN)
Trumpet (TR)Alto sax (AS)Clarinet (CL)
Fagotto (FG)
Marimba (MB)Vibraphone (VI)# of tones
1584
2304
1964
891
1080
1079
909
1332

degrades the accuracy of source separation. Spectral distances alsodegrade when pitch differences are 0, 1, and 11 when instruments
are identified incorrectly. However, when instruments are identified
correctly, spectral distances did not increase when many overtones
overlapped. This suggests that correct instrument identification enables precise source separation even when many overtones overlap.

Table 2. Constant values.
Parameter
ρH
ρI
λ
κ
LH
MH
LI
MI

Value
0.1
0.1
1.134
440.0
100
30
100
30

4. CONCLUSION

Table 3. Experimental results for instrument identification andsource separation, which show averaged log spectral distances in instrument sounds. Bold characters mean top two numbers.
Inst.
name
PF
VN
TR
AS
CL
FG
MB
VI
Avg.

Acc. rate [%]2 sounds
3 sounds
63.4
28.0
87.8
76.5
79.6
61.5
39.1
12.7
85.1
79.4
91.7
85.1
48.6
28.2
67.6
53.9
72.1
54.8

Log spect. dist. (×10−2 )2 sounds
3 sounds
3.25
3.88
2.43
3.31
2.78
3.40
3.29
3.97
1.59
2.12
1.83
2.39
5.25
5.34
6.43
5.86
3.12
3.65

We reported a method of simultaneously processing sound source
separation and musical instrument identification using Bayesian
spectral modeling. We defined the integrated harmonic and inharmonic tone models, decomposed the observed power spectrogramby using the expectation value of the latent variable, and identified the instrument for each sound in the audio mixture by selecting
the instrument based on maximum A Posteriori approximation. The
experimental results revealed that the accuracy of instrument identification and source separation rely on each other and correct instrument identification enables precise source separation even whenmany overtones overlap.
Acknowledgements: This research was partially supported by the
Ministry of Education, Science, Sports and Culture, Grant-in-Aid for
Scientific Research of Priority Areas, Primordial Knowledge Model
Core of Global COE program and JST CrestMuse Project.
5. REFERENCES

Fig. 2. Relationship between pitch (MIDI note number) differencein two instrument sounds to accuracy rate of instrument identification (left) and between the difference to log spectral distances ofseparated sounds whose instruments are correctly or incorrectly estimated (right).

It is easier to decompose audio mixture of two sounds than mix ofthree sounds and decreasing the number of sounds increases the accuracy rate of identification on average. This suggests precise source
separation increases the accuracy of instrument identification. The
marimba (MB) and vibraphone (VI) have larger spectral distances
than the other instruments. These instrument sounds have percussive properties and are sensitive to the diffusion of onset time. The
distances can decrease by accurately estimating the onset time.
Fig. 2 shows the relationship between the pitch difference in
two instrument sounds when two sounds are mixed to the accuracyof identification and the log spectral distance of separated sounds.
The pitch difference is based on the difference of MIDI pitch numbers. The spectral distances are shown in cases of correct and incorrect instrument identification. When pitch differences are 0 (unison), 1, 2, and 11, the Gaussian distribution for F0 overlaps withother harmonics and this overlap decreases the accuracy of identification. When the differences are 5 (perfect fourth) and 7 (perfectfifth), although the F0-Gaussian does not have any overlap, othermany harmonics have overlap and the overlap slightly decreases theidentification accuracy. This suggests that the overlap of overtones

3819[1] T. Kitahara, Computational Musical Instrument Recognition and Its
Application to Content-based Music Information Retrieval, Ph.D. thesis, Kyoto University, 2007.
[2] H. Kameoka et al., “A multipitch analyzer based on harmonic temporal
structured clustering,” IEEE Trans. Audio, Speech and Lang. Process.,
vol. 15, no. 3, pp. 982–994, 2007.
[3] K. Kashino, Computational Auditory Scene Analysis for Music Signals, Ph.D. thesis, University of Tokyo, 1994.
[4] M. Goto, “A real-time music-scene-analysis system: PredominantF0 estimation for detecting melody and bass lines in real-world audio
signals,” Speech Communication, vol. 43, no. 4, pp. 311–329, 2004.
[5] A. Klapuri, “Multipitch analysis of polyphonic music and speech signals using an auditory model,” IEEE Trans. Audio, Speech and Lang.
Process., vol. 16, no. 2, pp. 255–266, 2008.
[6] M. Goto, “An audio-based real-time beat tracking system for musicwith or without drum-sounds,” J. New Music Res., vol. 30, no. 2, pp.
159–171, 2001.
[7] S. Dixon, “Automatic extraction of tempo and beat from expressive
performances,” J. New Music Res., vol. 30, no. 1, pp. 39–58, 2001.
[8] M. E. P. Davies and M. D. Plumbley, “Context-dependent beat trackingof musical audio,” IEEE Trans. Audio, Speech and Lang. Process., vol.
15, no. 3, pp. 1009–1020, 2007.
[9] J. Woodruff et al., “Remixing stereo music with score-informed source
separation,” in ISMIR2006, pp. 314–319.
[10] H. Viste and G. Evangelista, “A method for separation of overlapping
partials based on similarity of temporal envelopes in multichannel mixtures,” IEEE Trans. Audio, Speech and Lang. Process., vol. 14, no. 3,
pp. 1051–1061, 2006.
[11] M. A. Casey and A. Westner, “Separation of mixed audio sources by
independent subspace analysis,” in ICMC2000, pp. 154–161.
[12] K. Yoshii et al., “Drum sound recognition for polyphonic audio signals
by adaptation and matching of spectrogram templates with harmonic
structure suppression,” IEEE Trans. Audio, Speech and Lang. Process.,
vol. 15, no. 1, pp. 333–345, 2007.
[13] H. Saruwatari et al., “Blind source separation combining independent
component analysis and beamforming,” EURASIP Journal on Applied
Signal Processing, vol. 2003, no. 11, pp. 1135–1146, 2003.
[14] K. Itoyama et al., “Integration and adaptation of harmonic and inharmonic models for separating polyphonic musical signals,” in
ICASSP2006, pp. 57–60.
[15] M. Goto et al., “RWC music database: Music genre database andmusical instrument sound database,” in ISMIR2003, pp. 229–230.