Proc. of the 12th Int. Conference on Digital Audio Effects (DAFx-09), Como, Italy, September 1-4, 2009

QUERY-BY-EXAMPLE MUSIC RETRIEVAL APPROACH BASED ON
MUSICAL GENRE SHIFT BY CHANGING INSTRUMENT VOLUME
Katsutoshi Itoyama†∗ , Masataka Goto‡ , Kazunori Komatani† , Tetsuya Ogata† , Hiroshi G. Okuno†† Graduate School of Informatics, Kyoto University* JSPS Research Fellow (DC1)‡ National Institute of Advanced Industrial Science and Technology (AIST)
itoyama [at] kuis.kyoto-u.ac.jp
ABSTRACTWe describe a novel Query-by-Example (QBE) approach inMusic Information Retrieval, which allows a user to customize query examples by directly modifying the volumeof different instrument parts. The underlying hypothesis isthat the musical genre shifts (changes) in relation to the volume balance of different instruments. On the basis of thishypothesis, we aim to clarify the relationship between thechange of the volume balance of a query and the shift in themusical genre of retrieved similar pieces, and thus help instruct a user in generating alternative queries without choosing other pieces. Our QBE system ﬁrst separates all instrument parts from the audio signal of a piece with the helpof its musical score, and then lets a user remix those partsto change acoustic features that represent musical mood of
the piece. The distribution of those features is modeled by
the Gaussian Mixture Model for each musical piece, and theEarth Movers Distance between mixtures of different pieces
is used as the degree of their mood similarity. Experimental results showed that the shift was actually caused by thevolume change of vocal, guitar, and drums.
1. INTRODUCTION
One of promising approaches of Music Information Retrieval is the Query-by-Example (QBE) retrieval [1, 2, 3,
4, 5, 6, 7] where a user can receive a list of musical pieces
ranked by their similarity to a musical piece (example) thatthe user gives as a query. Although this approach is powerful and useful, the user has to prepare or ﬁnd examples offavorite pieces and it was sometimes difﬁcult to control orchange the retrieved pieces after seeing them because another appropriate example should be found and given to getbetter results. For example, even if a user feels that vocal
or drum sounds are too strong in the retrieved pieces, it was
difﬁcult to ﬁnd another piece that has weaker vocal or drumsounds while keeping the basic mood and timbre of the ﬁrst
piece. Because ﬁnding such music pieces is a matter of trial
and error, we need more direct or convenient methods for
QBE.
We solve this inefﬁciency by allowing a user to create
new query examples for QBE by remixing existing musical pieces, i.e., changing the volume balance of the in-struments. To obtain the desired retrieved results, the user
can easily give alternative queries where the volume balance is changed from the original balance. For example, theabove problem can be solved by customizing a query example so that the volume of the vocal or drum sounds are
decreased. To remix an existing musical piece, we use an
original sound source separation method that decomposes
the audio signal of a musical piece into different instrument
parts on the basis of its available musical score. To measure
the similarity between the remixed query and each piece ina database, we use the Earth Movers Distance (EMD) between their Gaussian Mixture Models (GMMs). The GMM
for each piece is obtained by modeling the distribution oforiginal acoustic features consisting of intensity and timbrefeatures.
The underlying hypothesis is that changing the volume
balance of different instrument parts in a query causes themusical genre shift in the retrieved pieces — i.e., a user canchange the genre of the retrieved pieces just by changing
the volume balance of the query1 . An existing study [8] hasalready suggested this hypothesis. Based on this hypothesis, our research focuses on clarifying the relationship between the volume change of different instrument parts andthe shift in the musical genre in order to instruct a user ineasily generating alternative queries. To clarify this relationship, we conducted two different experiments. The ﬁrst
experiment examined how much change of the volume of asingle instrument part is needed to cause the musical genre
shift through our QBE retrieval system. The second experiment examined how the volume change of two instrument
parts (a two-instrument combination for the volume change)
affects the shift in the genre. This relationship is exploredby examining the genre distribution of the retrieved pieces.
Experimental results showed that the desired musical genre
shift in QBE results was easily caused just by changing thevolume balance of different instruments in the query.

1 We target genres which are mostly deﬁned by organization and volume
balance of musical instruments, such as classical music, jazz and rock. Although several genres are deﬁned by speciﬁc rhythm-patterns and singingstyle, e.g., a waltz and a hip-hop, we except them.

DAFX-1

Proc. of the 12th Int. Conference on Digital Audio Effects (DAFx-09), Como, Italy, September 1-4, 2009

Table 1: Acoustic features representing musical mood.
Acoustic intensity features
Dim.
Symbol Description
1
S1 (t)Overall intensity
2–8
S2 (i, t) Intensity of each subband*
Acoustic timbre features
Dim.
Symbol Description
9
S3 (t)
Spectral centroid
10
S4 (t)
Spectral width11
S5 (t)
Spectral rolloff
12
S6 (t)
Spectral ﬂux
13–19 S7 (i, t) Spectral peak of each subband*
20–26 S8 (i, t) Spectral valley of each subband*
27–33 S9 (i, t) Spectral contrast of each subband** 7-bands octave ﬁlterbank.

Figure 1: Overview of our QBE retrieval system basedon genre shift. Controlling the volume balance causes thegenre shift of a query song and our system returns songs
that are similar to the genre-shifted query.

2.2.1. Acoustic Intensity Features
Overall intensity for each frame, S1 (t), and intensity of eachsubband, S2 (i, t), are deﬁned as
S1 (t) =2. QUERY-BY-EXAMPLE RETRIEVAL BY
GENRE-SHIFTED MUSICAL PIECES

FN∑

X(t, f ) and S2 (i, t) =f =1

In this section, we describe our QBE retrieval system that
retrieves musical pieces based on the similarity of mood between musical pieces.
2.1. Musical Genre Shift
Our original term “musical genre shift” means the changeof the musical genre of pieces on auditory features which is
caused by changing the volume balance of musical instruments. For example, by boosting the vocal and reducing the
guitar and drums of a popular song, auditory features extracted from the modiﬁed song are similar to the featuresof a jazz song. The instrumentation and volume balanceof musical instruments are important in judging the musical genre. In fact, some musical genres are deﬁned by the
instrumentation. As shown in Figure 1, by automaticallyseparating the original recording (audio signal) of a piece
into musical instrument parts, a user can change their volume balance to cause the genre shift.
2.2. Acoustic Feature Extraction
Acoustic features that represent the musical mood are designed as shown in Table 1 upon existing studies of moodextraction [9]. These features extracted from the power
spectrogram, X(t, f ), for each frame (100 frames per second). The spectrogram is calculated by short-time Fouriertransform of the monauralized input audio signal, where t
and f are the frame and frequency indices, respectively.

∑

FH (i)X(t, f ),f =FL (i)where FN is the number of frequency bins of power spectrogram, FL (i) and FH (i) are the indices of lower and upperbounds for the i-th subband, respectively. The intensity ofeach subband helps to represent acoustic brightness. We use
octave ﬁlterbanks that divide the power spectrogram into noctave subbands:
[
) [
)
[
]
FN
FN FN
FN
1, n−1 , n−1 , n−2 , . . . ,, FN ,22
2
2
where n is the number of subbands and set to 7 in our experiments.
2.2.2. Acoustic Timbre Features
Acoustic timbre features consist of spectral shape featuresand spectral contrast features, which are known to be effective in detecting musical moods [10, 9]. The spectral shapefeatures are represented by spectral centroid S3 (t), spectral
width S4 (t), spectral rolloff S5 (t), and spectral ﬂux S6 (t)as follows:∑FN
f =1 X(t, f )f,
S3 (t) =S1 (t)∑FN2
f =1 X(t, f )(f − S3 (t))
S4 (t) =,S1 (t)DAFX-2∑S5 (t)X(t, f ) = 0.95S1 (t)

and

f =1

S6 (t) =

FN
∑
f =1

(log X(t, f ) − log X(t − 1, f ))2 .

Proc. of the 12th Int. Conference on Digital Audio Effects (DAFx-09), Como, Italy, September 1-4, 2009

On the other hand, the spectral contrast features are obtained as follows. Let a vector,

Table 2: Parameters of integrated tone model.
Symbol
(J)
wkl
(H)
(I)
wkl , wkl

(X(i, t, 1), X(i, t, 2), . . . , X(i, t, FN (i))),be the power spectrogram in the t-th frame and i-th subband. By sorting these elements in descending order, we
obtain another vector,

(H)

uklm

(X 0 (i, t, 1), X 0 (i, t, 2), . . . , X 0 (i, t, FN (i))),

(H)

vkln
(I)
uklm

where
X 0 (i, t, 1) > X 0 (i, t, 2) > · · · > X 0 (i, t, FN (i)),

(I)

vkln

and FN (i) is the number of the i-th subband frequency bins:
τkl
(H)
ρkl

FN (i) = FH (i) − FL (i).
Here, the spectral contrast features are represented by spectral peak S7 (i, t), spectral valley S8 (i, t), and spectral contrast S9 (i, t) as follows:( ∑βFN (i))
X 0 (i, t, f )f =1
S7 (i, t) = log,
βFN (i)∑
FN (i)
0f =(1−β)FN (i) X (i, t, f ) , and
S8 (i, t) = log βFN (i)
S9 (i, t) = S7 (i, t) − S8 (i, t),where β is a parameter to extract stable peak and valley values, and set to 0.2 in our experiments.
2.3. Similarity calculation
Our QBE retrieval system needs to calculate the similarity between musical pieces, i.e., a query example and eachpiece in a database, on the basis of the whole mood of thepiece. Musical genres usually depend on the whole mood,not detailed variations.
To model the mood of each piece, we use a Gaussian
Mixture Model (GMM) that approximates the distributionof acoustic features. We set the number of the mixtures to
8. Although the dimension of the obtained acoustic featureswas 33, it was reduced to 9 by using the principal component analysis where the cumulative percentage of eigenvalues was 0.95.
To measure the similarity among feature distributions,
we utilized Earth Movers Distance (EMD) [11]. The EMD
is based on the minimal cost needed to transform one distribution into another one.
3. SOUND SOURCE SEPARATION USINGINTEGRATED TONE MODEL
As mentioned in Section 1, musical audio signals should
be separated into instrument parts beforehand to boost andreduce the volume of those parts. We describe our soundsource separation method in this section.
Input and output of our method are described as follows:

(I)

ρkl

(H)

ωkl
(H)
σkl
β, κ

Description
Overall amplitude
Relative amplitude of harmonic and inharmonic
tone models
Amplitude coefﬁcient of temporal power envelope for harmonic tone model
Relative amplitude of n-th harmonic component
Amplitude coefﬁcient of temporal power envelope for inharmonic tone model
Relative amplitude of the n-th inharmonic component
Onset time
Diffusion of temporal power envelope for harmonic tone model
Diffusion of temporal power envelope for inharmonic tone model
F0 of harmonic tone model
Diffusion of harmonic components along frequency axis
Coefﬁcients that determine the arrangement of
the frequency structure of inharmonic model

Input Power spectrogram of a musical piece and its musical score (standard MIDI ﬁle).2 We suppose the spectrogram and the score have already been aligned (synchronized) by using another method.
Output Decomposed spectrograms that correspond to each
instrument.
To separate the power spectrogram, we approximate thepower spectrogram is additive. The musical audio signal
corresponding to the decomposed power spectrogram is obtained by using the inverse short-time Fourier transformwith the phase of the input spectrogram. In addition, weused an instrument sound database as training data for learning acoustic features of each musical instrument.
In this section, we ﬁrst deﬁne the problem of separating
sound sources and the integrated tone model. We then derive an iterative algorithm that consists of two steps: soundsource separation and model parameter estimation.
3.1. Integrated tone model of harmonic and inharmonicmodelsSeparating the sound source means decomposing the input
power spectrogram, X(t, f ), into a power spectrogram that
corresponds to each musical note, where t and f are the time
and the frequency, respectively. We assume that X(t, f ) includes K musical instruments and the k-th instrument performs Lk musical notes.
2 Standard MIDI ﬁles for famous songs are often available thanks to
Karaoke applications.

DAFX-3

Proc. of the 12th Int. Conference on Digital Audio Effects (DAFx-09), Como, Italy, September 1-4, 2009(a) Overview of harmonic tone model.

(b) Temporal structure of harmonic tone model. (c) Frequency structure of harmonic tone model.

Figure 2: Overall, temporal and frequency structures of harmonic tone model. Harmonic tone model consists of a twodimensional Gaussian Mixture Model, and is factorized into a pair of one-dimensional GMMs.
two-dimensional GMM that is a product of two one∑ (I) (I)∑ (I) (I)dimensional GMMs,
uklm Eklm (t) andvkln Fkln (f ).
(H)
The temporal structures of these tone models, Eklm (t) and(I)
Eklm (t), are deﬁned as an identical mathematical formula,(H)(I)
but the frequency structures, Fkln (f ) and Fkln (f ), are deﬁned as different forms. The deﬁnition of these models is
as follows:
Hkl (t, f ) =(a) Equally-spaced Gaussian ker- (b) Gaussian kernels obtained bynels along the log-scale frequency, changing the random variables ofF (f ).
the kernels in (a).

(H)

(H)

(H)

(H)

uklm Eklm (t)vkln Fkln (f ),m=0 n=1

Ikl (t, f ) =Figure 3: Frequency structure of inharmonic tone model.

M
NI
I −1 ∑
∑

(I)

(I)

(I)

(I)

uklm Eklm (t)vkln Fkln (f ),m=0 n=1((H) )(t − τklm )2
exp −
,
(H)
(H)
2πρkl
2(ρkl )2((H) )(f − ωkln )2
1(H)
Fkln (f ) = √
exp
−
,
(H)
(H)
2πσkl
2(σkl )2()
(I)(t − τklm )2
1(I)
Eklm (t) = √
exp −
,
(I)
(I)
2πρkl
2(ρkl )2()
1
(F(f ) − n)2(H)
Fkln (f ) = √
exp −
,
2
2π(f + κ) log β(H)Eklm (t) = √We use an integrated tone model, Jkl (t, f ), to represent the power spectrogram of the l-th musical note performed by the k-th musical instrument ((k, l)-th note). This
tone model is deﬁned as the sum of harmonic-structure tonemodels, Hkl (t, f ), and inharmonic-structure tone models,Ikl (t, f ), multiplied by the whole amplitude of the model,
(J)
wkl :
)
(J) ( (H)(I)
Jkl (t, f ) = wkl wkl Hkl (t, f ) + wkl Ikl (t, f ) ,( (H) (I) )(J)
where wkl and wkl , wkl satisﬁes the following constraints:∑ (J) ∫ ∫wkl =X(t, f ) dt df,

(H)

(H)

(H)

(H)

ωkln = nωkl ,
(I)

(I)

τklm = τkl + mρkl , and)/(f+ 1 log β.
F(f ) = log
κ(I)∀k, l : wkl + wkl = 1
The harmonic tone model, Hkl (t, f ), is deﬁned asa constrained two-dimensional Gaussian Mixture Model(GMM), which is a product of two one-dimensional GMMs,∑ (H) (H)
∑ (H) (H)uklm Eklm (t) andvkln Fkln (f ). This model is designed by referring to the harmonic-temporal-structured
clustering (HTC) source model [12]. Analogously, the inharmonic tone model, Ikl (t, f ), is deﬁned as a constrained1τklm = τkl + mρkl ,

k,l
(H)

M∑
NH
H −1 ∑All parameters of Jkl (t, f ) are listed in Table 2. Here, MH
and NH are the number of Gaussian kernels that represent temporal and frequency structures of the harmonic tonemodel, and MI and NI are the number of Gaussians that
represent the ones of the inharmonic tone model, respectively. β and κ are coefﬁcients that determine the arrangement of Gaussian kernels for the frequency structure of the

DAFX-4

Proc. of the 12th Int. Conference on Digital Audio Effects (DAFx-09), Como, Italy, September 1-4, 2009
(H)

(H)

(I)

(I)

inharmonic model3 . uklm , vkln , uklm , and vkln satisfy thefollowing conditions:
∑ (H)
∑ (H)
∀k, l :
uklm = 1, ∀k, l :vkln = 1,m∀k, l :∑

n(I)
uklm = 1, and∀k, l :∑ (I)vkln = 1.

m

n
(I)

As shown in Figure 3, function Fkln (f ) is derived bychanging the variables of the following probability density
function:()(g − n)21
N (g; n, 1) = √ exp −
,
2
2π

(I)

To evaluate the effectiveness of this separation, we use
an objective function deﬁned as the Kullback-Leibler (KL)(I)
(H)
divergence from Xklmn (t, f ) and Xklmn (t, f ) to the each
Gaussian kernel of the harmonic and inharmonic models:(∑ ∑ ∫∫(H)(∆)
Q=
Xklmn (t, f )
k,l

m,n

(H)

log

Xklmn (t, f )(H) (H)(H)(H)

uklm vkln Eklm (t)Fkln (f )∑ ∫∫(I)Xklmn (t, f )+m,nfrom g = F(f ) to f , i.e.,

log

)
dg (
(I)
N F(f ); n, 1
Fkln (f ) =(H)

Xklmn (t, f )(H) (H)

(H)

(H)

dt df

)
dt df .

uklm vkln Eklm (t)Fkln (f )

df

(
)
1
1
(F(f ) − n)2√ exp −=.
(f + κ) log β 2π23.2. Iterative separation algorithm
The goal of this separation is to decompose X(t, f ) into(k, l)-th note by multiplying a spectrogram distributionfunction, ∆(J) (k, l; t, f ), which satisﬁes∀k, l, t, f : 0 ≤ ∆(J) (k, l; t, f ) ≤ 1 and∑∆(J) (k, l; t, f ) = 1.
∀t, f :
k,lWith ∆(J) (k, l; t, f ), the separated power spectrogram,(J)
Xkl (t, f ), is obtained as(J)

Xkl (t, f ) = ∆(J) (k, l; t, f )X(t, f ).
Then, let ∆(H) (m, n; k, l, t, f ) and ∆(I) (m, n; k, l, t, f )be spectrogram distribution functions that decompose(J)
Xkl (t, f ) into each Gaussian distribution of the harmonicand inharmonic models, respectively. These functions satisfy∀k, l, m, n, t, f : 0 ≤ ∆(H) (m, n; k, l, t, f ) ≤ 1,
∀k, l, m, n, t, f : 0 ≤ ∆(I) (m, n; k, l, t, f ) ≤ 1,∑∆(H) (m, n; k, l, t, f )∀k, l, t, f :m,n∑∆(I) (m, n; k, l, t, f ) = 1.
+

and

m,n

With these functions, the separated power spectrograms,(H)(I)
Xklmn (t, f ) and Xklmn (t, f ), are obtained as
(H)

(J)

Xklmn (t, f ) = ∆(I) (m, n; k, l, t, f )Xkl (t, f ).

(J)

Xklmn (t, f ) = ∆(H) (m, n; k, l, t, f )Xkl (t, f )and

3 If 1/(log β) and κ are set to 1127 and 700, F (f ) is equivalent to themel scale of f Hz.

By minimizing Q(∆) pertaining to the distribution functions, we obtain the spectrogram distribution function anddecomposed spectrograms, i.e., separated sounds, on the basis of the parameters of the tone models.
Once the input spectrogram is decomposed, the likeliest
model parameters are calculated using a statistical estimation. We use prior distributions for several model parame(H)
(I)
(H)
(H)
(I)
(I)
ters, wkl and wkl , uklm , vkln , uklm , and vkln , to estimate
robust parameters. These prior distributions are deﬁned as
Dirichlet distributions. An another objective function, Q(θ) ,is used to estimate model parameters using the prior distributions, and is deﬁned as
(
)
(H)
(I)
(H)
(I)
Q(θ) =Q(∆) + log B wkl , wkl ; αw
,
α
wk
k
({
)
(H) } { (H) }+ log D uklm ; αukm({})(H) } {+ log D vkln ; αv(H)kn({)(I) } { (I) }+ log D uklm ; αukm({})(I) } {+ log D vkln ; αv(I).
knHere B(·) and D(·) are the probability density function(I)(H)
of the Beta and Dirichlet distributions, and αwk , αwk ,{ (H) } { (H) } { (I) }{ (I) }
αukm , αvkn , αukm , and αvkn are parameters ofthese distributions for each instrument. Note that this modiﬁcation of the objective function has no effect on the calculation of the distribution function. The parameter update
equations are described in Appendix A.
We obtain an iterative algorithm that consists of two
steps: one is the step in which the distribution function is
calculated while the model parameters are ﬁxed, and theother is the step in which the parameters are updated under
the distribution function. This iterative algorithm is equivalent to the Expectation-Maximization (EM) algorithm on
the basis of the maximum A Posteriori estimation.

DAFX-5

Proc. of the 12th Int. Conference on Digital Audio Effects (DAFx-09), Como, Italy, September 1-4, 2009(a) Genre shift caused by changing the volume of vocal.
Genre with the highest similarity changed from rock topopular and to jazz.

(a) Genre shift caused by changing the volume of vocal andguitar.

(b) Genre shift caused by changing the volume of guitar.
Genre with the highest similarity changed from rock topopular.

(b) Genre shift caused by changing the volume of vocal
and drums.

(c) Genre shift caused by changing the volume of guitarand drums.

(c) Genre shift caused by changing the volume of drums.
Genre with the highest similarity changed from popular torock and to dance.

Figure 4: Ratio of average EMD per genre to average EMD
of all genres while reducing or boosting the volume of single instrument part. Here, (a), (b), and (c) are for the vocal,
guitar, and drums, respectively. Note that a smaller ratio of
the EMD plotted in the lower area of the graph indicateshigher similarity.

Figure 5: Genres that have the smallest EMD (the highest
similarity) while reducing or boosting the volume of two instrument parts. The top, middle, and bottom are the casesof the vocal-guitar, vocal-drums, and guitar-drums, respectively.

DAFX-6

Proc. of the 12th Int. Conference on Digital Audio Effects (DAFx-09), Como, Italy, September 1-4, 2009

4. EXPERIMENTAL EVALUATION
We conducted experiments to explore the relationship between instrument volume balances and genres. Given
the query musical piece of which the volume balance ischanged, the genres of the retrieved musical pieces are investigated.
Ten musical pieces were excerpted for the query fromthe RWC Music Database: Popular Music (RWC-MDBP-2001 No. 1–10) [13]. The audio signals of these musical pieces were separated into each musical instrument partusing the standard MIDI ﬁles which are provided as the
AIST annotation [14]. The evaluation database consistedof other 50 musical pieces excerpted from the RWC Music
Database: Musical Genre (RWC-MDB-G-2001). This excerpted database includes musical pieces in the following
genres: Popular, Rock, Dance, Jazz, and Classical.
In the experiments, we reduced or boosted the volumesof three instrument parts — the vocal, guitar, and drums.
To shift the genre of the musical piece by changing the volume of these parts, the part of an instrument should havesufﬁcient duration4 . Thus, the above three instrument parts
were chosen because these parts satisfy the following two
constraints:
1. be played in all 10 musical pieces for the query, and
2. be played for more than 60 % of the duration of eachpiece.
The EMDs were calculated between the acoustic feature distributions of each query song and each piece in the
database, while reducing or boosting the volume of thesemusical instrument parts between −20 and +20 dB. Figure
4 shows the results of changing the volume of a single instrument part. The vertical axis is the relative ratio of the
EMD averaged over the 10 pieces, which is deﬁned as:
EMD ratio =average EMD of each genre.
average EMD of all genres

those songs actually have the rock taste with strong guitarand drum sounds.
By increasing the volume of the vocal from -20 dB, thegenre with the highest similarity shifted from rock (-20 to4 dB), to popular (5 to 9 dB), and to jazz (10 to 20 dB) as
shown in Figure 4 (a). By changing the volume of the guitar, the genre shifted from rock (-20 to 7 dB) to popular (8to 20 dB) as shown in Figure 4 (b). Although it was commonly observed that the genre shifted from rock to popularin both cases of vocal and guitar, the genre shift to jazz occurred only in the case of vocal. These results indicate thatthe vocal and guitar would have different importance in the
jazz music. By changing the volume of the drums, genres
shifted from popular (-20 to -7 dB), to rock (-6 to 4 dB),
and to dance (5 to 20 dB) as shown in Figure 4 (c). These
results indicate the reasonable relationship between the instrument volume balance and the musical genre shift, andthis relationship is consistent with typical images of musical genres.
4.2. Volume change of two instruments (pair)
The results in Figure 5 show that the genre shift depends
on the volume change of two instrument parts. If one ofthe part is not changed (at 0 dB), the results are same with
Figure 4.
Although the basic tendency in the genre shifts is similar
to the single instrument experiment, classical music, which
does not appear as the genre with the highest similarity in
Figure 4, appears in Figure 5 (b) when the vocal part isboosted and the drums part is reduced. The similarity ofrock music is decreased when we separately boosted one of
the guitar or drums, but it is interesting that rock music cankeep the highest similarity if both of the guitar and drums
are boosted together as shown in Figure 5 (c). This result
is well matched with the typical image of rock music, andsuggests promising possibilities as a tool for customizing
the query for QBE retrieval.
5. CONCLUSION

On the other hand, Figure 5 shows the results of simultaneously changing the volume of two instrument parts (instrument pairs).
4.1. Volume change of single instrument
The results in Figure 4 clearly show that the genre shift
occurred by changing the volume of any instrument part.
Note that the genre of the retrieved pieces at 0 dB (giving
the original queries without any changes) is same for all thethree ﬁgures (a), (b), and (c). Although we used 10 popularsongs excerpted from the RWC Music Database: PopularMusic for the queries, they are considered to be rock music at the genre with the highest similarity at 0 dB because
4 For example, the volume of an instrument that is performed for 5 seconds in a 5-minute musical piece may not affect the genre of the piece.

We have described musical genre shift by changing the volume of separated instrument parts and a QBE retrieval approach on the basis of the genre shift. This approach is important because it was not possible for a user to customize
the QBE query in the past and the user always had to ﬁnd
different pieces to obtain different retrieved results. By using the genre shift based on our original sound sourse separation method, it becomes easy and intuitive to customize
the QBE query just by changing the volume of instrument
parts. Experiments results conﬁrmed our hypothesis that themusical genre shifts in relation to the volume balance of instruments.
Although the current genre shift depends on only thevolume balance, other factors such as rhythm patterns,
sound effects, and chord progressions will also be useful
to cause the shift if we can control them. In the future, we

DAFX-7

Proc. of the 12th Int. Conference on Digital Audio Effects (DAFx-09), Como, Italy, September 1-4, 2009

plan to pursue the promising approach proposed in this paper and develop a better QBE retrieval system that easily
reﬂects user’s intention and preference.
6. ACKNOWLEDGEMENTS
This research was partially supported by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Scientiﬁc Research of Priority Areas, Primordial Knowledge Model Core of
Global COE program and JST CrestMuse Project.
7. REFERENCES[1] A. Rauber, E. Pampalk, and D. Merkl, “Using psychoacoustic models and self-organizing maps to create a hierarchical structuring of music by sound similarity,” in Proc.
ISMIR, 2002, pp. 71–80.
[2] C. Yang, “The macsis acoustic indexing framework for music retrieval: An experimental study,” in Proc. ISMIR, 2002,
pp. 53–62.
[3] E. Allamanche, J. Herre, O. Hellmuth, T. Kastner, and C. Ertel, “A multiple feature model for musical similarity retrieval,” in Proc. ISMIR, 2003.
[4] Y. Feng, Y. Zhuang, and Y. Pan, “Music information retrieval
by detecting mood via computational media aesthetics,” inProc. WI, 2003, pp. 235–241.
[5] B. Thoshkahna and K. R. Ramakrishnan, “Projekt quebex:
A query by example system for audio retrieval,” in Proc.
ICME, 2005, pp. 265–268.
[6] F. Vignoli and S. Pauws, “A music retrieval system basedon user-driven similarity and its evaluation,” in Proc. ISMIR,
2005, pp. 272–279.
[7] T. Kitahara, M. Goto, K. Komatani, T. Ogata, and H. G.
Okuno, “Musical instrument recognizer “instrogram” and its
application to music retrieval based on instrumentation similarity,” in Proc. ISM, 2006, pp. 265–274.
[8] K. Itoyama, M. Goto, K. Komatani, T. Ogata, and H.G.
Okuno, “Instrument equalizer for query-by-example retrieval: Improving sound source separation based on integrated harmonic and inharmonic models,” in Proc. ISMIR,
2008, pp. 133–138.
[9] L. Lu, D. Liu, and H. J. Zhang, “Automatic mood detection
and tracking of music audio signals,” IEEE Trans. Audio,
Speech and Lang. Process., vol. 14, no. 1, pp. 5–18, Jan.
2006.
[10] D. N. Jiang, L. Lu, H. J. Zhang, J. H. Tao, and L. H. Cai,
“Music type classiﬁcation by spectral contrast features,” inProc. ICME, 2002, pp. 113–116.
[11] Y. Rubner, C. Tomasi, and L. J. Guibas, “A metric for distributions with applications to image databases,” in Proc.
ICCV, 1998, pp. 59–66.
[12] H. Kameoka, T. Nishimoto, and S. Sagayama, “Harmonictemporal structured clustering via deterministic annealingem algorithm for audio feature extraction,” in Proc. ISMIR,
2005, pp. 115–122.
[13] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka,“RWC music database: Popular, classical, and jazz music
databases,” in Proc. ISMIR, 2002, pp. 287–288.
[14] M. Goto, “AIST annotation for the RWC music database,”in Proc. ISMIR, 2006, pp. 359–360.

A. PARAMETER UPDATE EQUATIONS
We describe the update equation for each parameter derived from
the M-step of the EM algorithm by differentiating the cost function(H)(I)
about each parameter. Let Xklmn (t, f ) and Xklmn (t, f ) be thedecomposed power:(H)(J)
Xklmn (t, f ) = ∆(H) (m, n; k, l, t, f )Xkl (t, f ) and
(I)

(J)

Xklmn (t, f ) = ∆(I) (m, n; k, l, t, f )Xkl (t, f ).
Summation or integration of the decomposed power over indices or variables are denoted by omitting these characters, e.g.,(H)(H)
Xkl (t, f ) and Xklm .
(J)

A.1. wkl : overall amplitude
(H)

(J)

(I)

wkl = Xkl + Xkl .
(I)

(H)

A.2. wkl , wkl : relative amplitude of harmonic and inharmonic tone models(H)(H)
Xkl + αwk(H)
wkl = (H)
and
(I)
(H)
(I)
Xkl + Xkl + αwk + αwk
(H)

wkl

(I)
(I)
Xkl + αwk
.
(H)
(I)
(I)
(H)
Xkl + Xkl + αwk + αwk=(H)

A.3. uklm : amplitude coefﬁcient of temporal power envelope
for harmonic tone model
(H)
(H)
X
+ αu
(H)
.
uklm = (H)klm P km
(H)
Xkl + m αukm
(H)

A.4. vkln : relative amplitude of n-th harmonic component
(H)

(H)

vkln =(H)

Xkln + αvkn
P (H) .
(H)
Xkl + n αvkn

(I)

A.5. uklm : amplitude coefﬁcient of temporal power envelope
for inharmonic tone model
(I)
(I)
X
+ αu
(I)
.
uklm = (I)klm P km
(I)
Xkl + m αukm
(I)

A.6. vkln : relative amplitude of n-th inharmonic component
(I)

(I)

vkln =(I)

Xkln + αvkn
P (I) .
(I)
Xkl + n αvkn

A.7. τkl : onset time
XZ
(H)
(H)
τkl =
(t − mρkl )Xklm (t) df
m

+
‹`

XZ

(I)
(I)
(t − mρkl )Xklm (t) df

!

m

(I) ´
(H)
Xkl + Xkl .

(H)

A.8. ωkl : F0 of harmonic tone modelP R(H)
nf Xkln (f ) df
(H)
n
ωkl =
.
P 2 (H)n n Xkln(H)

A.9. σkl : diffusion of harmonic components along frequency
axis
v
uP R
(H)
(H)
u
(f − nωkl )2 Xkln (f ) df
(H)
σkl = t n
.
(H)
Xkl

DAFX-8