Hindawi Publishing Corporation
EURASIP Journal on Advances in Signal Processing
Volume 2010, Article ID 172961, 14 pages
doi:10.1155/2010/172961

Research Article
Query-by-Example Music Information Retrieval by
Score-Informed Source Separation and Remixing Technologies
Katsutoshi Itoyama,1 Masataka Goto,2 Kazunori Komatani,1 Tetsuya Ogata,1and Hiroshi G. Okuno1
1 Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University, Sakyo-Ku,

Kyoto 606-8501, Japan
2 Media Interaction Group, Information Technology Research Institute (ITRI), National Institute of Advanced Industrial Science andTechnology (AIST), Tsukuba, Ibaraki 305-8568, Japan
Correspondence should be addressed to Katsutoshi Itoyama, itoyama@kuis.kyoto-u.ac.jp
Received 1 March 2010; Revised 10 September 2010; Accepted 31 December 2010
Academic Editor: Augusto Sarti
Copyright © 2010 Katsutoshi Itoyama et al. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly
cited.
We describe a novel query-by-example (QBE) approach in music information retrieval that allows a user to customize query
examples by directly modifying the volume of diﬀerent instrument parts. The underlying hypothesis of this approach is thatthe musical mood of retrieved results changes in relation to the volume balance of diﬀerent instruments. On the basis of thishypothesis, we aim to clarify the relationship between the change in the volume balance of a query and the genre of the retrieved
pieces, called genre classification shift. Such an understanding would allow us to instruct users in how to generate alternative queries
without finding other appropriate pieces. Our QBE system first separates all instrument parts from the audio signal of a piece withthe help of its musical score, and then it allows users remix these parts to change the acoustic features that represent the musicalmood of the piece. Experimental results showed that the genre classification shift was actually caused by the volume change in the
vocal, guitar, and drum parts.

1. Introduction
One of the most promising approaches in music information
retrieval is query-by-example (QBE) retrieval [1–7], wherea user can receive a list of musical pieces ranked by theirsimilarity to a musical piece (example) that the user gives asa query. This approach is powerful and useful, but the user
has to prepare or find examples of favorite pieces, and it issometimes diﬃcult to control or change the retrieved pieces
after seeing them because another appropriate example
should be found and given to get better results. For example,
even if a user feels that vocal or drum sounds are too strong
in the retrieved pieces, it is diﬃcult to find another piecethat has weaker vocal or drum sounds while maintaining thebasic mood and timbre of the first piece. Since finding such
music pieces is now a matter of trial and error, we need more
direct and convenient methods for QBE. Here we assume thatQBE retrieval system takes audio inputs and treat low-level
acoustic features (e.g., Mel-frequency cepstral coeﬃcients,
spectral gradient, etc.).
We solve this ineﬃciency by allowing a user to create newquery examples for QBE by remixing existing musical pieces,that is, changing the volume balance of the instruments. To
obtain the desired retrieved results, the user can easily givealternative queries by changing the volume balance fromthe piece’s original balance. For example, the above problem
can be solved by customizing a query example so that thevolume of the vocal or drum sounds is decreased. To remix
an existing musical piece, we use an original sound source
separation method that decomposes the audio signal of amusical piece into diﬀerent instrument parts on the basisof its musical score. To measure the similarity between theremixed query and each piece in a database, we use the EarthMovers Distance (EMD) between their Gaussian Mixture

2
Models (GMMs). The GMM for each piece is obtained bymodeling the distribution of the original acoustic features,
which consist of intensity and timbre.
The underlying hypothesis is that changing the volume
balance of diﬀerent instrument parts in a query grows
diversity of the retrieved pieces. To confirm this hypothesis,
we focus on the musical genre since musical diversity andmusical genre have a certain level of relationship. A music
database that consists of various genre pieces is suitable forthe purpose. We define the term genre classification shift as
the change of musical genres in the retrieved pieces. We
target genres that are mostly defined by organization andvolume balance of musical instruments, such as classicalmusic, jazz, and rock. We exclude genres that are definedby specific rhythm patterns and singing style, e.g., waltz andhip hop. Note that this does not mean that the genre of thequery piece itself can be changed. Based on this hypothesis,
our research focuses on clarifying the relationship between
the volume change of diﬀerent instrument parts and theshift in the musical genre of retrieved pieces in order
to instruct a user in how to easily generate alternative
queries. To clarify this relationship, we conducted three
diﬀerent experiments. The first experiment examined how
much change in the volume of a single instrument part isneeded to cause a genre classification shift using our QBE
retrieval system. The second experiment examined how thevolume change of two instrument parts (a two-instrument
combination for volume change) cooperatively aﬀects theshift in genre classification. This relationship is exploredby examining the genre distribution of the retrieved pieces.
These experimental results show that the desired genre
classification shift in the QBE results was easily achieved bysimply changing the volume balance of diﬀerent instrumentsin the query. The third experiment examined how thesource separation performance aﬀects the shift. The retrieved
pieces using sounds separated by our method are comparedwith those using original sounds before mixing down in
producing musical pieces. The experimental result showedthat the separation performance for predictable feature shifts
depends on an instrument part.

2. Query-by-Example Retrieval by
Remixed Musical Audio Signals
In this section, we describe our QBE retrieval system for
retrieving musical pieces based on the similarity of mood
between musical pieces.

EURASIP Journal on Advances in Signal Processing
genre classification shift in our QBE approach suggests thatremixing query examples grow the diversity of retrieved
results. As shown in Figure 1, by automatically separating
the original recording (audio signal) of a piece into musicalinstrument parts, a user can change the volume balance of
these parts to cause a genre classification shift.
2.2. Acoustic Feature Extraction. Acoustic features that represent the musical mood are designed as shown in Table 1
upon existing studies of mood extraction [8]. These features
extracted from the power spectrogram, X(t, f ), for eachframe (100 frames per second). The spectrogram is calculated by short-time Fourier transform of the monauralized
input audio signal, where t and f are the frame andfrequency indices, respectively.
2.2.1. Acoustic Intensity Features. Overall intensity for eachframe, S1 (t), and intensity of each subband, S2 (i, t), aredefined as
S1 (t) =

FN





f =1





X t, f ,(1)

f =FL (i)where FN is the number of frequency bins of the power
spectrogram and FL (i) and FH (i) are the indices of lower andupper bounds for the ith subband, respectively. The intensityof each subband helps to represent acoustic brightness. We
use octave filter banks that divide the power spectrogram into
n octave subbands:


 

1,







F
F
FN
F
, nN−1 , nN−2 , . . . , N , FN ,
2n−1
2
2
2

(2)

where n is the number of subbands, which is set to 7 inour experiments. These filter banks cannot be constructedbecause they have ideal frequency response; we implemented
these by division and sum of the power spectrogram.
2.2.2. Acoustic Timbre Features. Acoustic timbre featuresconsist of spectral shape features and spectral contrast
features, which are known to be eﬀective in detecting musicalmoods [8, 9]. The spectral shape features are represented byspectral centroid S3 (t), spectral width S4 (t), spectral rolloﬀ
S5 (t), and spectral flux S6 (t):FN

f =1 XS3 (t) =2.1. Genre Classification Shift. Our original term “genre
classification shift” means a change in the musical genreof pieces based on auditory features, which is caused bychanging the volume balance of musical instruments. For
example, by boosting the vocal and reducing the guitar anddrums of a popular song, auditory features are extracted
from the modified song are similar to the features of a jazzsong. The instrumentation and volume balance of musicalinstruments aﬀects the musical mood. The musical genre
does not have direct relation to the musical mood but

F
H (i)

S2 (i, t) =X t, f ,FN

f =1 XS4 (t) =
S
5 (t)







t, f f
,
(t)
S1





2

t, f f − S3 (t),S1 (t)(3)

X t, f = 0.95S1 (t),f =1

S6 (t) =

FN









log X t, f − log X t − 1, f

f =1

2

.

EURASIP Journal on Advances in Signal Processing

3

Original
recording

A popular song

Popular songs

QBE-MIRsystemDrums

VolumeSound source separation
Popular songs

Popular-like mix(same as the original)

Dr. Gt. Vo.
Dance songsSoundsourceDr. Gt. Vo.
Jazz songs

Jazz-like mix
Volume

Vocal

VolumeDance-like mix
Guitar

Dr. Gt. Vo.
Mixdown

Retrievalresults

Re-mixed
recordings

Volume balance
control by users

Genre-shifted
queriesFigure 1: Overview of QBE retrieval system based on genre classification shift. Controlling the volume balance causes a genre classification
shift of a query song, and our system returns songs that are similar to the genre-shifted query.

Table 1: Acoustic features representing musical mood.
Acoustic intensity features
SymbolS1 (t)
S2 (i, t)
Acoustic timbre features
SymbolS3 (t)
S4 (t)
S5 (t)
S6 (t)
S7 (i, t)
S8 (i, t)
S9 (i, t)

Dim.
1
2–8
Dim.
9
10
11
12
13–19
20–26
27–33
∗

Description
Overall intensity
Intensity of each subband∗
Description
Spectral centroid
Spectral width
Spectral rolloﬀ
Spectral flux
Spectral peak of each subband∗
Spectral valley of each subband∗
Spectral contrast of each subband∗

7-band octave filter bank.

The spectral contrast features are obtained as follows. Leta vector,(X(i, t, 1), X(i, t, 2), . . . , X(i, t, FN (i))),

(4)

where
X  (i, t, 1) > X  (i, t, 2) > · · · > X  (i, t, FN (i))(6)

be the power spectrogram in the tth frame and ith subband.
By sorting these elements in descending order, we obtain
another vector,

as shown in Figure 3 and FN (i) is the number of the ith
subband frequency bins:(X  (i, t, 1), X  (i, t, 2), . . . , X  (i, t, FN (i))),

FN (i) = FH (i) − FL (i).

(5)

(7)

4

EURASIP Journal on Advances in Signal Processing(a) −∞ dB(b) −5 dB(c) ±0 dB(d) +5 dB(e) +∞ dB

Figure 2: Distributions of the first and second principal components of extracted features from the no. 1 piece of the RWC Music Database:
Popular Music. Five figures show the shift of feature distribution by changing the volume of the drum part. The shift of feature distribution
causes the genre classification shift.

(X  (i, t, 1), . . . , X  (i, t, FN (i)))

X(i, t, 1)
X(i, t, 2)
X(i, t, 3)X  (i, t, 1)
X  (i, t, 2)
X  (i, t, 3)

Sort

Power

(X(i, t, 1), . . . , X(i, t, FN (i)))

Power

Power
spectrogram

Index

Frequency index

Figure 3: Sorted vector of power spectrogram.

Here, the spectral contrast features are represented byspectral peak S7 (i, t), spectral valley S8 (i, t), and spectral
contrast S9 (i, t):⎛ βFN (i)X i, t, f⎠,
βFN (i)

f =1

S7 (i, t) = log⎝

⎛ F (i)N

S8 (i, t) = log⎝

⎞




f =(1−β)FN (i) X

i, t, fβFN (i)

⎞
⎠,

(8)

S9 (i, t) = S7 (i, t) − S8 (i, t),where β is a parameter for extracting stable peak and valley
values, which is set to 0.2 in our experiments.
2.3. Similarity Calculation. Our QBE retrieval system needs
to calculate the similarity between musical pieces, that is, aquery example and each piece in a database, on the basis of
the overall mood of the piece.
To model the mood of each piece, we use a Gaussian
Mixture Model (GMM) that approximates the distributionof acoustic features. We set the number of mixtures to 8
empirically, although a previous study [8] used a GMM with
16 mixtures since we used smaller database than that study
for experimental evaluation. Although the dimension of theobtained acoustic features was 33, it was reduced to 9 byusing the principal component analysis where the cumulative
percentage of eigenvalues was 0.95.
To measure the similarity among feature distributions,
we utilized Earth Movers Distance (EMD) [10]. The EMD
is based on the minimal cost needed to transform one
distribution into another one.

3. Sound Source Separation Using
Integrated Tone Model
As mentioned in Section 1, musical audio signals should
be separated into instrument parts beforehand to boost
and reduce the volume of those parts. Although a numberof sound source separation methods [11–14] have beenstudied, most of them still focus on dealing with music
performed on either pitched instruments that have harmonicsounds or drums that have inharmonic sounds. For example,
most separation methods for harmonic sounds [11–14]
cannot separate inharmonic sounds, while most separation
methods for inharmonic sounds, such as drums [15], cannot
separate harmonic ones. Sound source separation methods
based on the stochastic properties of audio signals, forexample, independent component analysis and sparse coding[16–18], treat particular kind of audio signals which arerecorded with a microphone array or have small numberof simultaneously voiced musical notes. However, thesemethods cannot separate complex audio signals such ascommercial CD recordings. We describe our sound source
separation method which can separate complex audio signals
with both harmonic and inharmonic sounds in this section.
The input and output of our method are described asfollows:
input power spectrogram of a musical piece and its
musical score (standard MIDI file); standard MIDI
files for famous songs are often available thanks to
Karaoke applications; we assume the spectrogram
and the score have already been aligned (synchronized) by using another method;
output decomposed spectrograms that correspond toeach instrument.

EURASIP Journal on Advances in Signal Processing

5

To separate the power spectrogram, we approximate thepower spectrogram which is purely additive. By playing backeach track of the SMF on a MIDI sound module, we prepareda sampled sound for each note. We call this a template soundand used it as prior information (and initial values) in theseparation. The musical audio signal corresponding to thedecomposed power spectrogram is obtained by using theinverse short-time Fourier transform with the phase of the
input spectrogram.
In this section, we first define the problem of separating
sound sources and the integrated tone model. This modelis based on a previous study [19], and we improved
implementation of the inharmonic models. We then derive
an iterative algorithm that consists of two steps: sound source
separation and model parameter estimation.

The definitions of these models are as follows:MNH
H −1 



Hkl t, f = 

m=0 n=1



Ikl t, f =MNII −1 
m=0 n=1(H)(H) (H)u(H)klm Eklm (t)vkln Fkln f ,

 

(I)
(I) (I)u(I)klm Eklm (t)vkln Fkln f ,

⎛

2 ⎞

⎛

2 ⎞

(H)
1
⎜ t − τklm ⎟(H)
Eklm (t) = √ ⎠,(H) exp⎝−
(H) 2
2πρkl
2 ρkl
(H)
f − ωkln
1
⎜
⎟
f =√
exp
⎝−
 ⎠,
(H) 2
2πσkl(H)2 σkl(H)  Fkln⎛

3.1. Integrated Tone Model of Harmonic and Inharmonic Models. Separating the sound source means decomposing the
input power spectrogram, X(t, f ), into a power spectrogramthat corresponds to each musical note, where t and f are thetime and the frequency, respectively. We assume that X(t, f )includes K musical instruments and the kth instrument
performs Lk musical notes.
We use an integrated tone model, Jkl (t, f ), to represent
the power spectrogram of the lth musical note performed by
the kth musical instrument ((k, l)th note). This tone model
is defined as the sum of harmonic-structure tone models,
Hkl (t, f ), and inharmonic-structure tone models, Ikl (t, f ),multiplied by the whole amplitude of the model, wkl(J) :










Jkl t, f = wkl(J) wkl(H) Hkl t, f + wkl(I) Ikl t, f



,

(9)

where wkl(J) and (wkl(H) , wkl(I) ) satisfy the following constraints:

wkl(J) =

X t, f dt df ,

(H)

(I)

∀k, l : wkl + wkl = 1. (10)k,lThe harmonic tone model, Hkl (t, f ), is defined asa constrained two-dimensional Gaussian Mixture Model(GMM), which is a product of two one-dimensional GMMs,
 (H) (H)
 (H) (H)uklm Eklm (t) and vkln
Fkln ( f ). This model is designedby referring to the HTC source model [20]. Analogously,
the inharmonic tone model, Ikl (t, f ), is defined as a constrained two-dimensional GMM that is a product of two (I) (I)(I)
vkln Fkln ( f ).
one-dimensional GMMs, u(I)klm Eklm (t) and(H)
The temporal structures of these tone models, Eklm(t) and(I)
Eklm (t), are defined as an identical mathematical formula,(H)(I)
but the frequency structures, Fkln
( f ) and Fkln
( f ), aredefined as diﬀerent forms. In the previous study [19], theinharmonic models are implemented in a nonparametricway. We changed the inharmonic model by implementing
in a parametric way. This change improves generalization of
the integrated tone model, for example, timbre modeling andextension to a bayesian estimation.

2 ⎞
⎟
2 ⎠,

(11)

(I)
t − τklm

1⎜(I) exp⎝−
(I)
2πρkl
2 ρkl

(I)
(t) = √
Eklm

 

 

F f −n1(I)  

exp −
Fkln
f =√ 
2
2π f + κ log β

2 

,

(H)
(H)
τklm
= τkl + mρkl ,(H)(H)= nωkl ,
ωkln
(I)
(I)
= τkl + mρkl ,
τklm

 

F f

=

log







f /κ + 1.
log βAll parameters of Jkl (t, f ) are listed in Table 2. Here, MH and
NH are the numbers of Gaussian kernels that represent temporal and frequency structures of the harmonic tone model,
respectively, and MI and NI are the numbers of Gaussians
that represent those of the inharmonic tone model. β and κ
are coeﬃcients that determine the arrangement of Gaussian
kernels for the frequency structure of the inharmonic model.
If 1/(log β) and κ are set to 1127 and 700, F ( f ) is equivalent
(H)
(I)
(I)
to the mel scale of f Hz. Moreover u(H)klm , vkln , uklm , and vkln
satisfy the following conditions:∀k, l : (H)m∀k, l : (H)n∀k, l :

vkln = 1,
(12)

 (I)
m

∀k, l :

uklm = 1,

uklm = 1,

 (I)nvkln = 1.

(I)
As shown in Figure 5, function Fkln
( f ) is derived bychanging the variables of the following probability density
function:

 



g −n
1
N g; n, 1 = √ exp −
2
2π

2 

,

(13)

6

EURASIP Journal on Advances in Signal Processing∑ (H) (H)muklm Eklm (t)

Power

Time

Fre
qu

enc
y
∑ (H) (H)nvkln Fkln ( f )(a) overview of harmonic tone model(H) (H)

vkl1 Fkl1 ( f )∑ (H) (H)muklm Eklm (t)(H) (H)vkl2 Fkl2 ( f )(H) (H)(H) (H)vkl3 Fkl3 ( f )
Power

Power

ukl0 Ekl0 (t)(H) (H)
ukl1 Ekl1 (t)(H) (H)(H)

σkl

ukl2 Ekl2 (t)

τkl

(H)

ωkl

ρ(H)
kl
Time

(H)

2ωkl

(H)

3ωkl

Frequency(b) temporal structure of harmonic tone model(c) frequency structure of harmonic tone model

Power

Power

Figure 4: Overall, temporal, and frequency structures of the harmonic tone model. This model consists of a two-dimensional Gaussian
Mixture Model, and it is factorized into a pair of one-dimensional GMMs.

1

2

3

7

8

F (f)
(I)
gn ( f ) = vkln N (F ( f ); n, 1)
g7 ( f )g1 ( f )g8 ( f )g2 ( f )Sum of theseg3 ( f )(a) Equally-spaced Gaussian kernels along the log-scale frequency,
F ( f ).

F −1 (1) F −1 (2)F −1 (3) F −1 (7) F −1 (8)f(I)
Hn ( f ) ∝ (vkln /( f + k))N (F ( f ); n, 1)
H1 ( f )H7 ( f )H2 ( f )H8 ( f )Sum of theseH3 ( f )(b) Gaussian kernels obtained by changing the random variablesof the kernels in (a).

Figure 5: Frequency structure of inharmonic tone model.

EURASIP Journal on Advances in Signal Processing

7

Table 2: Parameters of integrated tone model.
Symbolwkl(J)
wkl(H) , wkl(I)
u(H)
klm
(H)
vkln
u(I)
klm
(I)
vkln
τkl
ρkl(H)
ρkl(I)
ωkl(H)
σkl(H)β, κDescription
Overall amplitude
Relative amplitude of harmonic and inharmonic tone models
Amplitude coeﬃcient of temporal power envelope for harmonic tone model
Relative amplitude of the nth harmonic component
Amplitude coeﬃcient of temporal power envelope for inharmonic tone model
Relative amplitude of the nth inharmonic component
Onset time
Diﬀusion of temporal power envelope for harmonic tone modelDiﬀusion of temporal power envelope for inharmonic tone model
F0 of harmonic tone modelDiﬀusion of harmonic components along frequency axis
Coeﬃcients that determine the arrangement of the frequency structure of inharmonic modelfrom g = F ( f ) to f , that is,
 

(I)
f =
Fkln

With these functions, the separated power spectrograms,(H)(I)(t, f ) and Xklmn(t, f ), are obtained as
Xklmn

dg   
N F f ; n, 1
df

























(H)
t, f = Δ(H) m, n; k, l, t, f Xkl(J) t, f ,
Xklmn

 

 

F f −n1
1
√ exp −= 
f + κ log β 2π
2

(14)

2 

.

3.2. Iterative Separation Algorithm. The goal of this separation is to decompose X(t, f ) into each (k, l)th note by multiplying a spectrogram distribution function, Δ(J) (k, l; t, f ),that satisfies
To evaluate the eﬀectiveness of this separation, we use
an objective function defined as the Kullback-Leibler (KL)(H)
(I)
divergence from Xklmn(t, f ) and Xklmn(t, f ) to each Gaussian
kernel of the harmonic and inharmonic models:Q∀k, l, t, f : 0 ≤ Δ k, l; t, f ≤ 1,
∀t, f : Δ(J) k, l; t, f = 1.
(J)

(Δ)

⎛
   (H) 

= ⎝
X
t, f
klmn

m,n

k,l



(15)

× log

k,l

+

With Δ(J) (k, l; t, f ), the separated power spectrogram,
Xkl(J) (t, f ), is obtained as






 



Xkl(J) t, f = Δ(J) k, l; t, f X t, f .

∀k, l, m, n, t, f : 0 ≤ Δ(H) m, n; k, l, t, f ≤ 1,
∀k, l, m, n, t, f : 0 ≤ Δ(I) m, n; k, l, t, f ≤ 1,
∀k, l, t, f : 0 ≤ Δ(H) m, n; k, l, t, f

(17)

× log

+

Δ

m,n

(I)





m, n; k, l, t, f = 1.

  dt df(H) (H)(H)u(H)
klm vkln Eklm (t)Fkln f

(20)


(I) 
Xklmn
t, f


(I)
t, f
Xklmn

⎞



  dt df ⎠.

(I) (I)(I)u(I)
klm vkln Eklm (t)Fkln f

The spectrogram distribution functions are calculated byminimizing Q(Δ) for the functions. Since the functions
satisfy the constraint given by (18), we use the method of
Lagrange multiplier. Since Q(Δ) is a convex function forthe spectrogram distribution functions, we first solve thesimulteneous equations, that is, derivatives of the sum of Q(Δ)
and Lagrange multipliers for condition (18) are equal to zero,and then obtain the spectrogram distribution functions,
Δ

(H)

(18)
Δ

(I)

 

(H)
E(H) (t)Fkln
f

 ,

m, n; k, l, t, f = klm
Jt,fk,l kl



m,n



(H)
t, f
Xklmn

 
m,n

(16)

Then, let Δ(H) (m, n; k, l, t, f ) and Δ(I) (m, n; k, l, t, f ) be spectrogram distribution functions that decompose Xkl(J) (t, f )into each Gaussian distribution of the harmonic and inharmonic models, respectively. These functions satisfy

(19)

(I)
t, f = Δ(I) m, n; k, l, t, f Xkl(J) t, f .
Xklmn



 

(I)
(I)
(t)Fkln
Eklm
f

 ,
m, n; k, l, t, f = k,l Jkl t, f





(21)

8

EURASIP Journal on Advances in Signal Processing

and decomposed spectrograms, that is, separated sounds, on
the basis of the parameters of the tone models.
Once the input spectrogram is decomposed, the likeliest model parameters are calculated using a statisticalestimation. We use auxiliary objective functions for each(Y ), to estimate robust parameters with power(k, l)th note, Qk,lspectrogram of the template sounds, Ykl (t, f ). The (k, l)th
auxiliary objective function is defined as the KL divergence(H)(I)(t, f ) and Yklmn(t, f ) to each Gaussian kernel offrom Yklmn
the harmonic and inharmonic models:
(Y )
=
Qk,l

 
m,n

+


(H) 
Yklmn
t, f log

 
m,n





(H)
t, f
Yklmn



  dt df(H) (H)(H)u(H)
klm vkln Eklm (t)Fkln f



(I)
t, f
Yklmn



(I)
Yklmn
t, f log  dt df ,(I) (I)(I)u(I)
klm vkln Eklm (t)Fkln f

(22)
where

























(H)
Yklmn
t, f = Δ(H) m, n; k, l, t, f Ykl t, f ,(I)t, f = Δ(I) m, n; k, l, t, f Ykl t, f .
Yklmn

(23)

Then, let Q be a modified objective function that is defined(Y )as the weighted sum of Q(Δ) and Qk,lwith weight parameterα:
Q = αQ(Δ) + (1 − α)



(Y )
Qk,l
.

(24)

k,l

We can prevent the overtraining of the models by gradually
increasing α from 0 (i.e., the estimated model should firstbe close to the template spectrogram) through the iterationof the separation and adaptation (model estimation). The
parameter update equations are derived by minimizing Q.
We experimentally set α to 0.0, 0.25, 0.5, 0.75, and 1.0 insequence and 50 iterations are suﬃcient for parameter convergence with each alpha value. Note that this modificationof the objective function has no direct eﬀect on the calculation of the distribution functions since the modification
never changes the relationship between the model and the
distribution function in the objective function. For all α
values, the optimal distribution functions are calculated fromonly the models written in (21). Since the model parameters
are changed by the modification, the distribution functions
are also changed indirectly. The parameter update equations
are described in the appendix.
We obtain an iterative algorithm that consists of two
steps: calculating the distribution function while the model
parameters are fixed and updating the parameters under the
distribution function. This iterative algorithm is equivalent
to the Expectation-Maximization (EM) algorithm on thebasis of the maximum a posteriori estimation. This fact
ensures the local convergence of the model parameter
estimation.

4. Experimental Evaluation
We conducted two experiments to explore the relationshipbetween instrument volume balances and genres. Given the

Table 3: Number of musical pieces for each genre.
Genre
Popular
Rock
Dance
Jazz
Classical

Number of pieces
6
6
15
9
14

query musical piece in which the volume balance is changed,
the genres of the retrieved musical pieces are investigated.
Furthermore, we conducted an experiment to explore theinfluence of the source separation performance on thisrelationship, by comparing the retrieved musical pieces
using clean audio signals before mixing down (original) andseparated signals (separated).
Ten musical pieces were excerpted for the query fromthe RWC Music Database: Popular Music (RWC-MDB-P2001 no. 1–10) [21]. The audio signals of these musicalpieces were separated into each musical instrument partusing the standard MIDI files, which are provided as the
AIST annotation [22]. The evaluation database consistedof 50 other musical pieces excerpted from the RWCMusic Database: Musical Genre (RWC-MDB-G-2001). This
excerpted database includes musical pieces in the following
genres: popular, rock, dance, jazz, and classical. The numberof pieces are listed in Table 3.
In the experiments, we reduced or boosted the volumesof three instrument parts—vocal, guitar, and drums. To shift
the genre of the retrieved musical piece by changing thevolume of these parts, the part of an instrument shouldhave suﬃcient duration. For example, the volume of an
instrument that is performed for 5 seconds in a 5-minutemusical piece may not aﬀect the genre of the piece. Thus,
the above three instrument parts were chosen because theysatisfy the following two constraints:(1) played in all 10 musical pieces for the query,(2) played for more than 60% of the duration of eachpiece.
At http://winnie.kuis.kyoto-u.ac.jp/∼itoyama/qbe/, sound examples of remixed signals and retrieved results areavailable.
4.1. Volume Change of Single Instrument. The EMDs were
calculated between the acoustic feature distributions of eachquery song and each piece in the database as describedin Section 2.3, while reducing or boosting the volume ofthese musical instrument parts between 20 and +20 dB.
Figure 6 shows the results of changing the volume of a single
instrument part. The vertical axis is the relative ratio of the
EMD averaged over the 10 pieces, which is defined as
EMD ratio =average EMD of each genre.
average EMD of all genres(25)

The results in Figure 6 clearly show that the genre
classification shift occurred by changing the volume of any

EMD ratio

Dissimilar

EURASIP Journal on Advances in Signal Processing

9

1.3
1.2
1.1
1

Similar

0.9
0.8
Rock
0.7

−20

Popular

−10

0

Jazz
10

20

Volume control ratio of vocal part (dB)
Popular
Rock
Dance

Jazz
Classical

1.3

1.3
EMD ratio Dissimilar

EMD ratio

Dissimilar(a) genre classification shift caused by changing the volume of vocal. Genrewith the highest similarity changed from rock to popular and to jazz

1.2
1.1
1

0.8
Popular

Rock
0.7

−20

1.1
1
0.9

Similar

Similar

0.9

1.2

0
10
−10
Volume control ratio of guitar part (dB)
Popular
Rock
Dance

0.8
Popular
0.7

20

Jazz
Classical(b) genre classification shift caused by changing the volume of guitar. Genrewith the highest similarity changed from rock to popular

−20

Rock

Dance

0
10
−10
Volume control ratio of drums part (dB)
Popular
Rock
Dance

20

Jazz
Classical(c) genre classification shift caused by changing the volume of drums. Genrewith the highest similarity changed from popular to rock and to dance

Figure 6: Ratio of average EMD per genre to average EMD of all genres while reducing or boosting the volume of single instrument part.
Here, (a), (b), and (c) are for the vocal, guitar, and drums, respectively. Note that a smaller ratio of the EMD plotted in the lower area of
the graph indicates higher similarity. (a) Genre classification shift caused by changing the volume of vocal. Genre with the highest similarity
changed from rock to popular and to jazz. (b) Genre classification shift caused by changing the volume of guitar. Genre with the highest
similarity changed from rock to popular. (c) Genre classification shift caused by changing the volume of drums. Genre with the highest
similarity changed from popular to rock and to dance.

instrument part. Note that the genre of the retrieved pieces
at 0 dB (giving the original queries without any changes) is
the same for all three Figures 6(a), 6(b), and 6(c). Although
we used 10 popular songs excerpted from the RWC Music
Database: Popular Music for the queries, they are considered
to be rock music as the genre with the highest similarity at
0 dB because those songs actually have the true rock flavor
with strong guitar and drum sounds.
By increasing the volume of the vocal from −20 dB, thegenre with the highest similarity shifted from rock (−20 to4 dB) to popular (5 to 9 dB) and to jazz (10 to 20 dB) as
shown in Figure 6(a). By changing the volume of the guitar,
the genre shifted from rock (−20 to 7 dB) to popular (8 to20 dB) as shown in Figure 6(b). Although it was commonly
observed that the genre shifted from rock to popular in bothcases of vocal and guitar, the genre shifted to jazz only in thecase of vocal. These results indicate that the vocal and guitar
would have diﬀerentimportance in jazz music. By changing
the volume of the drums, genres shifted from popular (−20to −7 dB) to rock (−6 to 4 dB) and to dance (5 to 20 dB)

10EURASIP Journal on Advances in Signal Processing

Volume control ratio of vocal part (dB)

20

10

0

−10

−20
−20

−10

0

10

20

Volume control ratio of guitar part (dB)(a) genre classification shift caused by changing the volume of vocal andguitar20

Volume control ratio of guitar part (dB)Volume control ratio of vocal part (dB)

20

10

0

−10

−20
−20

−10

0
10
Volume control ratio of drums part (dB)20(b) genre classification shift caused by changing the volume of vocal and
drums

10

0

−10

−20
−20

−10

0

10

20

Volume control ratio of drums part (dB)(c) genre classification shift caused by changing the volume of guitar anddrums

Figure 7: Genres that have the smallest EMD (the highest similarity) while reducing or boosting the volume of two instrument parts. (a),(b), and (c) are the cases of the vocal-guitar, vocal-drums, and guitar-drums, respectively. (a) Genre classification shift caused by changing
the volume of vocal and guitar. (b) Genre classification shift caused by changing the volume of vocal and drums. (c) Genre classification shift
caused by changing the volume of guitar and drums.

as shown in Figure 6(c). These results indicate a reasonable
relationship between the instrument volume balance and thegenre classification shift, and this relationship is consistent
with typical impressions of musical genres.
4.2. Volume Change of Two Instruments (Pair). The EMDs
were calculated in the same way as the previous experiment.

Figure 7 shows the results of simultaneously changing thevolume of two instrument parts (instrument pairs). If oneof the parts is not changed (at 0 dB), the results are the sameas those in Figure 6.
Although the basic tendency in the genre classification
shifts is similar to the single instrument experiment, classicalmusic, which does not appear as the genre with the highest

EURASIP Journal on Advances in Signal Processing

11

Dissimilar

0.3

Shifted EMD

0.2
0.1
0

Similar

−0.1

−0.2
−0.3

−5

0
5
10
Volume control ratio of vocal part (dB)15

Separated
Popular
Dance
Jazz

Original
PopularDance
Jazz(a) normalized EMDs by changing the volume of vocal

0.1
0

0.1
0

−0.1

Similar−0.1

Similar

0.2

Shifted EMD

Shifted EMD

0.2

−0.2
−0.3

0.3

Dissimilar

Dissimilar

0.3

−5

0
5
10
Volume control ratio of vocal part (dB)

Original
Popular
Dance
Jazz

15

Separated
PopularDance
Jazz(b) normalized EMDs by changing the volume of guitar

−0.2
−0.3

−5

0
5
10
Volume control ratio of vocal part (dB)

Original
Popular
Dance
Jazz

15

Separated
PopularDance
Jazz(c) normalized EMDs by changing the volume of drums

Figure 8: Normalized EMDs that are shifted to 0 when the volume control ratio is 0 dB for each genre while reducing or boosting thevolume. (a), (b), and (c) graphs are obtained by changing the volume of the vocal, guitar, and drum parts, respectively. Note that a smaller
EMD plotted in the lower area of each graph indicates higher similarity than the one without volume controlling. (a) Normalized EMDs bychanging the volume of vocal. (b) Normalized EMDs by changing the volume of guitar. (c) Normalized EMDs by changing the volume ofdrums.

similarity in Figure 6, appears in Figure 7(b) when the vocal
part is boosted and the drum part is reduced. The similarityof rock music decreased when we separately boosted either
the guitar or the drums, but it is interesting that rock music
can keep the highest similarity if both the guitar and drums
are boosted together as shown in Figure 7(c). This resultclosely matched with the typical impression of rock music,
and it suggests promising possibilities for this technique as atool for customizing the query for QBE retrieval.
4.3. Comparison between Original and Separated Sounds. The
EMDs were calculated while reducing or boosting the volumeof the musical instrument parts between −5 and +15 dB.
Figure 8 shows the normalized EMDs that are shifted to 0
when the volume control ratio is 0 dB. Since all query songs

are popular music, EMDs between query songs and popular
pieces in the evaluation database tend to be smaller thanthe pieces of other genres. In this experiment, EMDs werenormalized because we focused on the shifts in the acoustic
features.
By changing the volume of the drums, the EMDs plottedin Figure 8(c) have similar curves in both of the originaland separated conditions. On the other hand, by changing
the volume of the guitar, the EMDs plotted in Figure 8(b)showed that a curve of the original condition is diﬀerent froma curve of the separation condition. This result indicatesthat the shifts of features in those conditions were diﬀerent.
Average source separation performance of the guitar partwas −1.77 dB, which was a lower value than those of vocal
and drum parts. Noises included in the separated sounds

12

EURASIP Journal on Advances in Signal Processingof the guitar part induced this diﬀerence. By changing thevolume of the vocal, the plotted EMDs of popular anddance pieces have similar curves, but the EMDs of jazzpieces have diﬀerent curves, although the average source
separation performance of the vocal part is the highest
among these three instrument parts. This result indicatesthat the separation performance for predictable feature shifts
depends on the instrument part.

5. Discussions
The aim of this paper is achieving a QBE approach which
can retrieve diverse musical pieces by boosting or reducing
the volume balance of the instruments. To confirm theperformance of the QBE approach, evaluation using a music
database which has wide variations is necessary. A music
database that consists of various genre pieces is suitable forthe purpose. We defined the term genre classification shift as
the change of musical genres in the retrieved pieces since wefocus on the diversity of the retrieved pieces not on musicalgenre change of the query example.
Although we conducted objective experiments to evaluate the eﬀectiveness of our QBE approach, several questions
remain as open questions.
(1) More evidences of our QBE approach by subjective
experiments are needed whether the QBE retrieval
system can help users search better results.
(2) In our experiments, we used only popular musicalpieces as query examples. Remixing query examples
except popular pieces can shift genres of retrieved
results.
For source separation, we use the MIDI representationof a musical signal. Mixed and separated musical signals
contain variable features: timbre diﬀerence from musicalinstruments’ individuality, characteristic performances of
instrument players such as vibrato, and environments suchas room reverberation and sound eﬀects. These features
can be controlled implicitly by changing the volume ofmusical instruments and therefore QBE systems can retrieve
various musical pieces. Since MIDI representations do notcontain these features, diversity of retrieved musical pieces
will decrease and users cannot evaluate the mood diﬀerenceof the pieces if we use only musical signals which aresynthesized from MIDI representations.
In the experiments, we used precisely synchronized
SMFs at most 50 milliseconds of onset timing error. In
general, synchronization between CD recordings and their
MIDI representations is not enough for separation. Previous
studies on audio-to-MIDI synchronization methods [23, 24]
can help this problem. We experimentally confirmed that
onset timing error under 200 milliseconds does not decrease
source separation performance. Another problem is thatthe proposed separation method needs a complete musicalscore with melody and accompaniment instruments. A studyof source separation method with a MIDI representationof specified instrument part [25] will help solving theaccompaniment problem.

In this paper, we aimed to analyze and decompose a mixture of harmonic and inharmonic sounds by appending theinharmonic model to the harmonic model. To achieve this,a requirement must be satisfied: one-to-one basis-source
mapping based on structured and parameterized source
model. The HTC source model [20], on which our integrated
model is based, satisfies the requirement. Adaptive harmonic
spectral decomposition [26] has modeled a harmonic structure in a diﬀerent way. They are suitable for multiple-pitch
analysis and applied to polyphonic music transcription. On
the other hand, the nonnegative matrix factorization (NMF)
is usually used for separating musical instrument sounds
and extracting simple repeating patterns [27, 28] and only
approximates complex audio mixture since the one-to-one
mapping is uncertified. Eﬃcient feature extraction fromcomplex audio mixtures will be promising by combining
lower-order analysis using structured models such as the
HTC and higher-order analysis using unconstrained models
such as the NMF.

6. Conclusions
We have described how musical genres of retrieved pieces
shift by changing the volume of separated instrument partsand explained a QBE retrieval approach on the basis ofsuch genre classification shift. This approach is importantbecause it was not possible for a user to customize the QBE
query in the past, which required the user to always finddiﬀerent pieces to obtain diﬀerent retrieved results. By using
the genre classification shift based on our original soundsource separation method, it becomes easy and intuitive tocustomize the QBE query by simply changing the volume
of instrument parts. Experimental results confirmed ourhypothesis that the musical genre shifts in relation to thevolume balance of instruments.
Although the current genre shift depends on only thevolume balance, other factors such as rhythm patterns, soundeﬀects, and chord progressions would also be useful forcausing the shift if we could control them. In the future,
we plan to pursue the promising approach proposed in thispaper and develop a better QBE retrieval system that easilyreflects the user’s intention and preferences.

Appendix
Parameter Update Equations
The update equation for each parameter derived from theM-step of the EM algorithm is described here. We solved
the simultaneous equations, that is, derivatives of the sum of
the cost function (24), and Lagrange multipliers for model
parameter constraints, (10) and (12), are equal to zero. Here
we introduce the weighted sum of decomposed powers:






 







Zkl t, f = αΔ(J) k, l; t, f X t, f + (1 − α)Ykl t, f ,
























(H)
t, f = Δ(H) m, n; k, l, t, f Zkl t, f ,
Zklmn
(I)
t, f = Δ(I) m, n; k, l, t, f Zkl t, f .
Zklmn

(A.1)

EURASIP Journal on Advances in Signal Processing

13

The summation or integration of the decomposed powerover indices, variables, and suﬃxes is denoted by omittingthese characters, for example,
(H) 

Zkl



t, f =



(H) 




n



⎜n

(A.3)

wkl(H) and wkl(I) are the relative amplitude of harmonic andinharmonic tone models:Zkl(H) + Zkl(I)

(A.4)

.

u(H)
klm is the amplitude coeﬃcient of temporal power envelope
for harmonic tone model:
u(H)
klm =

(H)
Zklm

.

Zkl(H)(A.5)(H)
is the relative amplitude of the nth harmonic compovkln
nent:
(H)
=
vkln

(H)
Zkln

Zkl(H)

.

(A.6)

u(I)
kln is the amplitude coeﬃcient of temporal power envelope
for inharmonic tone model:
(I)
Zklm

u(I)
klm =

Zkl(I)

.

(A.7)

(I)
is the relative amplitude of the nth inharmonic compovkln
nent:
(I)
=
vkln

τkl is the onset time:


τkl =

m

(I)
Zkln

Zkl(I)

(H)(H)(t)dt+t − mρkl
Zklm

.

(A.8)


m



(I)
(I)
(t)dt
t − mρkl
Zklm

Zkl(H) +Zkl(I)(A.9)(H) (H)ωkl is the F0 of harmonic tone model:
ωkl


n
(H)
=
ωkl

 

(H)
n f Zkln
f df


n

(H)
n2 Zkln⎞1/2

Zkl(H)⎟⎠.

(A.11)

This research was partially supported by the Ministry of
Education, Science, Sports and Culture, a Grant-in-Aid
for Scientific Research of Priority Areas, the Primordial
Knowledge Model Core of Global COE program, and the JST
CrestMuse Project.

References

Zkl(H),(H)
Zkl + Zkl(I)
Zkl(I)

 

Acknowledgmentswkl(J) = Zkl(H) + Zkl(I) .

wkl(I) =

2

(H)
(H)
f − nωkl
Zkln
f df ⎟

(A.2)



(H)
Zklmn
t, f df .

wkl(J) is the overall amplitude:wkl(H) =⎛ σkl(H) = ⎜⎝Zklmn t, f ,

m,n

(H)
(t) =
Zklm

σkl(H) is the diﬀusion of harmonic components along frequency axis:,(A.10)[1] A. Rauber, E. Pampalk, and D. Merkl, “Using psycho-acoustic
models and self-organizingmaps to create a hierarchicalstructuring of music bysound similarity,” in Proceedings ofthe International Conference on Music Information Retrieval(ISMIR ’02), pp. 71–80, 2002.
[2] C. C. Yang, “The MACSIS acoustic indexingframework formusic retrieval: an experimental study,” in Proceedings ofthe International Conference on Music Information Retrieval(ISMIR ’02), pp. 53–62, 2002.
[3] E. Allamanche, J. Herre, O. Hellmuth, T. Kastner, and
C. Ertel, “A multiple feature model for musical similarity
retrieval,” in Proceedings of the International Conference onMusic Information Retrieval (ISMIR ’03), pp. 217–218, 2003.
[4] Y. Feng, Y. Zhuang, and Y. Pan, “Music information retrieval
by detecting mood viacomputational media aesthetics,” inProceedings of the International Conference on Web Intelligence(WI ’03), pp. 235–241, 2003.
[5] B. Thoshkahna and K. R. Ramakrishnan, “Projektquebex: aquery by example system for audioretrieval,” in Proceedings ofthe International Conference on Multimedia and Expo (ICME’05), pp. 265–268, 2005.
[6] F. Vignoli and S. Pauws, “A music retrievalsystem based onuser-driven similarity and its evaluation,” in Proceedings ofthe International Conference on Music Information Retrieval(ISMIR ’05), pp. 272–279, 2005.
[7] T. Kitahara, M. Goto, K. Komatani, T. Ogata, and H. G.
Okuno, “Musical instrument recognizer “instrogram” andits application to music retrieval based on instrumentation
similarity,” in Proceedings of the Annual International Supply
Management Conference (ISM ’06), pp. 265–274, 2006.
[8] L. Lu, D. Liu, and H. J. Zhang, “Automatic mood detection andtracking of music audio signals,” IEEE Transactions on Audio,
Speech and Language Processing, vol. 14, no. 1, pp. 5–18, 2006.
[9] D.-N. Jiang, L. Lu, H.-J. Zhang, J.-H. Tao, and L.-H. Cai,
“Music type classification by spectral contrast features,” inProceedings of the International Conference on Multimedia andExpo (ICME ’02), pp. 113–116, 2002.
[10] Y. Rubner, C. Tomasi, and L. J. Guibas, “A metric for distributions with applications to image databases,” in Proceedings ofthe International Conference On Computer Vision (ICCV ’98),
pp. 59–66, 1998.
[11] T. Virtanen and A. Klapuri, “Separation of harmonic sounds
using linear models for the overtone series,” in Proceedings ofthe International Conference on Acoustics, Speech and Signal
Processing (ICASSP ’02), vol. 2, pp. 1757–1760, 2002.

14[12] M. R. Every and J. E. Szymanski, “A spectralfiltering approach
to music signal separation,” in Proceedings of the Conference on Digital Audio Eﬀects (DAFx ’04), pp. 197–200,
2004.
[13] J. Woodruﬀ, P. Pardo, and R. Dannenberg, “Remixing stereo
music with score-informed source separation,” in Proceedingsof the International Conference on Music Information Retrieval(ISMIR ’06), pp. 314–319, 2006.
[14] H. Viste and G. Evangelista, “A method for separation of
overlapping partials based on similarity of temporal envelopes
in multichannel mixtures,” IEEE Transactions on Audio, Speechand Language Processing, vol. 14, no. 3, pp. 1051–1061,
2006.
[15] D. Barry, D. Fitzgerald, E. Coyle, and B. Lawlor, “Drum source
separation using percussive feature detection and spectral
modulation,” in Proceedings of the Irish Signals and Systems
Conference (ISSC ’05), pp. 13–17, 2005.
[16] H. Saruwatari, S. Kurita, K. Takeda, F. Itakura, T. Nishikawa,
and K. Shikano, “Blind source separation combining independent component analysis and beamforming,” EURASIP
Journal on Applied Signal Processing, vol. 2003, no. 11, pp.
1135–1146, 2003.
[17] M. A. Casey and A. Westner, “Separation of mixed audio
sources by independent subspace analysis,” in Proceedings ofthe International Computer Music Conference (ICMC ’00), pp.
154–161, 2000.
[18] M. D. Plumbley, S. A. Abdallah, J. P. Bello, M. E. Davies, G.
Monti, and M. B. Sandler, “Automatic music transcription andaudio source separation,” Cybernetics and Systems, vol. 33, no.
6, pp. 603–627, 2002.
[19] K. Itoyama, M. Goto, K. Komatani, T. Ogata, and H. G.
Okuno, “Integration and adaptation of harmonic and inharmonic models for separating polyphonic musical signals,” inProceedings of the International Conference on Acoustics, Speechand Signal Processing (ICASSP ’07), pp. 57–60, 2007.
[20] H. Kameoka, T. Nishimoto, and S. Sagayama, “A multipitch
analyzer based on harmonic temporal structured clustering,”
IEEE Transactions on Audio, Speech and Language Processing,vol. 15, no. 3, pp. 982–994, 2007.
[21] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC
music database: popular, classical, and jazz music databases,”in Proceedings of the International Conference on Music Information Retrieval (ISMIR ’02), pp. 287–288, 2002.
[22] M. Goto, “AIST annotation for the RWC music database,” inProceedings of the International Conference on Music Information Retrieval (ISMIR ’06), pp. 359–360, 2006.
[23] R. J. Turetsky and D. P. W. Ellis, “Groundtruth transcriptionsof real music from force-aligned MIDI synthesis,” in Proceedings of the International Conference on Music Information
Retrieval (ISMIR ’03), 2003.
[24] M. Muller, Information Retrieval for Musicand Motion, chapter
5, Springer, Berlin, Germany, 2007.
[25] N. Yasuraoka, T. Abe, K. Itoyama, K. Komatani, T. Ogata,and G. Hiroshi, “Changing timbre and phrase in existingmusical performances as you like,” in Proceedings of the ACM
International Conference on Multimedia (ACM-MM ’09), pp.
203–212, 2009.
[26] E. Vincent, N. Bertin, and R. Badeau, “Adaptive harmonic
spectral decomposition for multiple pitch estimation,” IEEE
Transactions on Audio, Speech and Language Processing, vol. 18,
no. 3, pp. 528–537, 2010.

EURASIP Journal on Advances in Signal Processing[27] M. N. Schmidt and M. Mørup, “Nonnegative matrix factor 2D deconvolution for blind single channel source separation,”in Proceedings of the International Workshop on Independent
Component Analysis and Signal Separation (ICA ’06), pp. 700–
707, April 2006.
[28] P. Smaragdis, “Convolutive speech bases and their application
to supervised speech separation,” IEEE Transactions on Audio,
Speech and Language Processing, vol. 15, no. 1, pp. 1–12, 2007.