COMPARATIVE EVALUATIONS OF VARIOUS HARMONIC/PERCUSSIVE SOUNDSEPARATION ALGORITHMS BASED ON ANISOTROPIC CONTINUITY OF
SPECTROGRAM
Hideyuki Tachibana1 , Hirokazu Kameoka1,2 , Nobutaka Ono3 , Shigeki Sagayama1
1

Graduate School of Information Science and Technology, The University of Tokyo,
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan
2
NTT Communication Science Laboratories, NTT Corporation,
3-1 Morinosato Wakamiya, Atsugi, Kanagawa 243-0198, Japan
3
National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan
ABSTRACTIn this paper, we explore several algorithms to ﬁnd the best performing algorithm for harmonic and percussive sound separation(HPSS) based on anisotropic continuity of spectrogram throughcomparative evaluation of their experimental performance. Separating harmonic and percussive sounds is useful as a preprocessor for
many music analysis purposes including chord estimation, rhythm
analysis, and other music information retrieval tasks. We haveintroduced a method called ”Harmonic/Percussive Sound Separation” (HPSS), that decomposes a music signal into two components
by separating the spectrogram into horizontally-continuous andvertically-continuous components, which roughly correspond toharmonic and percussive sounds, respectively. Many possible ways
exist to realize the HPSS algorithm based on this concept while
it has been unknown which algorithm performs best. This paper
describes the details of ﬁve different HPSS algorithms and compares
their performances over real music signals.
Index Terms— Harmonic/percussive sound separation, Music
information retrieval, Source Separation
1. INTRODUCTION
Music signals are typically composed of harmonic components (suchas violin and guitar) and percussive components (such as snare andbass drums), which have very different properties. This paper addresses methods for separating harmonic and percussive components
in music signals. Such methods can be used as powerful preprocessors for many music-information-retrieval (MIR) tasks, including
chord estimation, melody extraction, and rhythm pattern recognition. They can also be used for music listening applications, whichallows the users to control the volumes of drum and other sounds
separately. Owing to its potential usefulness, many attempts havebeen made to develop such methods in the music signal processing
area (e.g., [1]).
So far, we have introduced a method called Harmonic/Percussive
Sound Separation (HPSS). The key point of this method is that it
focuses on the difference in the directions of continuity between the
spectrograms of harmonic and percussive components. Speciﬁcally,
the spectrograms of harmonic components are typically continuous
in the time direction, owing to their quasi-stationarity, while theThis work was supported by JSPS Grand-In-Aid for research fellows(22-6961).

978-1-4673-0046-9/12/$26.00 ©2012 IEEE

465

(a)

(b)

(c)

Fig. 1. Concept of HPSS. (a) Spectrogram of the input signal W ,(b) harmonic component H , which should be continuous in time, (c)percussive component P , which should be continuous in frequency.

spectrograms of percussive components are typically continuous
in the frequency direction, owing to their impulsiveness. We haveveriﬁed that HPSS has been effective as a preprocessor for several
MIR applications [2], including audio chord estimation [3], rhythm
map generation [4], and audio melody extraction [5, 6].
There are many ways to formulate HPSS algorithms based on
the concept of the anisotropic continuous of spectrogram, and eachalgorithm has its own parameters that need to be tuned. Of all thesealgorithms, we have not yet veriﬁed which one performs best interms of separation performance. We also have not fully investigatedwhich values of the tuning parameters provide the best performancefor each of the algorithms. As HPSS can be used as an effective preprocessor for many MIR tasks, it is important to investigate which
HPSS variant performs best. This paper describes the details of each
HPSS algorithms, and provides comparative evaluation of the devised algorithms using real music signals.
2. FRAMEWORK OF HARMONIC/PERCUSSIVE SOUNDSEPARATIONIn this section, we describe the general framework of HPSS. The
method is designed to decompose an input signal w(t) into harmonic component h(t) and percussive component p(t) by a processing on spectrogram domain. That is, given a power spectrogram
W = STFT[w(t)] = (Wt,k )1≤t≤T,1≤k≤K (Fig. 1 (a)), HPSS estimates the spectrogram H (Fig. 1 (b)) and P (Fig. 1 (c)), which arecontinuous in time and in frequency respectively. Generally, HPSSis formulated as a problem to ﬁnd H and P that satisfy following
properties:

ICASSP 2012

(A) Spectrogram H should be continuous in time, and P should
be continuous in frequency.

J1 (H , P ; γ, κ) = ΩH (H γ ) + κΩP (P γ )(4)(B) Sum of H and P should be approximately equal to the original spectrogram W .

J2 (H , P ; γ, a, b) = ΩH (H γ ) + aΩP (P γ )X+b(Wt,k γ − Ht,k γ − Pt,k γ )2 .

(5)(C) Each component of the spectrogram should be non-negative,i.e., ∀(τ, k), Hτ,k ≥ 0, Pτ,k ≥ 0.
There could be principally two approached to obtain spectrograms H and P that satisfy these properties. An approach is to simply extract the horizontal lines and the vertical lines from spectrograms, by applying image-processing techniques, such as low-passﬁltering in time and frequency directions respectively. The other approach is to obtain the power spectrograms by optimizing a criterion
that measure (A) above, under the condition that the constraints (B)and (C) should be satisﬁed. Details of those methods are describedin the following sections.
After estimating the power spectrograms, we should apply timefrequency masking to estimate the waveform h(t) and p(t) from theestimated power spectrograms H and P . In this procedure, we also
have a choice what type of time-frequency mask to apply. In thispaper, we consider following 3 choices: (a) we do not apply any
time-frequency masks, but calculate the amplitude from estimatedpower spectrogram directly, which will be referred to as “None,”(b) estimate the amplitude by Wiener masking, and (c) by binary
masking.

t,k2
2
J3 (H , P ; γ, σH , σP ) = ΩH (H γ )/σH+ ΩP (P γ )/σP
+ DI (W |H + P )Fig. 2. Three types of objective functions of HPSS, derived from thecontinuity criteria (A) and the constraints (B) and (C) described in
section 2.
Table 1. List of the parameters and their candidate values.
ParameterValue
γ in Eq. 1, 4, 5
0.3, 0.5, 1.0
κ in Eq. 4 for HM1
0.25, 0.5, 0.6, 0.9, 1, 1.2, 1.5, 2, 4
κ in Eq. 4 for HM2
0.8 + 0.04n, (n = 1 · · · 9)
a in Eq. 5
0.5, 1, 2
b in Eq. 5 and c in Eq. 2 0.25, 0.5, 0.6, 0.9, 1, 1.2, 1.5, 2, 4
σH , σP in Eq. 6
0.1, 0.3, 0.5
I, J in Eq. 2, 3
3, 10, 20
Frame length L
256, 512, 1024 points (16 kHz)Time-frequency maskNone, Wiener, Binary

3. FORMULATIONS OF HPSS
3.1. Power Spectrogram Estimation based on Low-Pass Filtering of Spectrogram
As mentioned in the previous section, the simplest way to achieve theconcept of HPSS is to extract the horizontal and vertical lines fromthe spectrogram by applying low-pass ﬁltering, and regard them H
and P respectively as follows,
X
X
Wt+τ,k γ f (τ ), Pt,k γ =Wt,k+κ γ g(κ), (1)Ht,k γ =τκwhere f (τ ), g(κ) are low-pass ﬁlters, and γ is an exponential factor
to suppress the effects of relatively strong components, and the value
is preferable to be around 0.3 to 0.5. Hereafter, we call this algorithm
as “2DF.”As to thus derived power spectrograms, it is conceivable thatmost percussive components are suppressed in Ht,k , because percussive components are instantaneous and it should easily be ﬁltered out
by low-pass ﬁltering. Similarly, most harmonic components should
be suppressed in Pt,k . The forms of f and g are arbitrary, but in thispaper we used the following forms for simplicity:f (τ ) = 1 − cos(2πτ /I), − I ≤ τ ≤ I,
(2)g(κ) = c{1 − cos(2πκ/J)}, − J ≤ κ ≤ J,
(3)
where I, J and c are constants to be tuned.
3.2. Power Spectrogram Estimation based on Optimization Approach
Power-spectrogram-estimation by optimizing the continuity criteria
is also an effective approach. The objective functions of HPSS wehave considered are listed in Fig. 2. In this section, we describe theexplicit form and the design concepts of each term of the objectivefunctions, and show the way to solve the optimization problems.
All objective functions in Fig. 2 have terms ΩH (H γ ) andΩP (P γ ). These terms reﬂect the concept (A), i.e., they are the(6)continuity criterion of the spectrogram H and P in the respective
directions. Although there could be many types of speciﬁc formof the continuity criterion of the spectrogram, one of the simplest
form is the norm of the differences between the neighboring bins ofspectrogram. That is, the continuity of spectrogram in time and in
frequency can be evaluated by the following criteria respectively.
T
−1 X
K
X
(Hτ +1,k γ − Hτ,k γ )2 ,
(7)
ΩH (H γ ) =τ =0 k=0

ΩP (P γ )

=

T K−1
X
X

(Pτ,k+1 γ − Pτ,k γ )2 .

(8)

τ =0 k=0

3.2.1. Optimization under Hard Mixing Constraint
Eq. 4 in Fig. 2 consists of only the continuity criterion (A). In thiscase, inconveniently, the solution that optimize the criterion is trivial,
and the sum of the spectrograms does not sufﬁce the concept (B).
Therefore, we had to make constraints on the spectrogram based on
the concept (B). One of the simplest constraint is the following one,which is based on the assumption that the spectrogram is additive,i.e., Ht,k ξ +Pt,k ξ −Wt,k ξ = 0, where ξ is an exponential factor. Forξ, we consider two feasible parameter settings in this paper, ξ = γ
and ξ = 2γ.
When ξ = γ, the objective function J1 (H , P ; γ, κ) can be minimized by iterating the following formulae [7],Hτ,k γPτ,k γwhere β =←←min(max(Hτ,k γ + β/4, 0), Wτ,k γ ),Wτ,k γ − Hτ,k γ ,

(9)
(10)

(1 + κ)−1 {(Hτ +1,k γ − 2Hτ,k γ + Hτ −1,k γ )−κ(Pτ,k+1 γ − 2Pτ,k γ + Pτ,k−1 γ )}.
(11)Hereafter, we refer to this algorithm as “HM1.” As for γ and ξ, if
γ = ξ = 0.5 is satisﬁed, the condition h(t) + p(t) = w(t) is466

Method
HM1
HM2
SE
Idiv
2DF

Parameter set that maximizes SDR of H
L = 512, γ = 0.3,
κ = 1, Wiener
L = 256, γ = 1,
κ = 0.92, None
L = 512, a = 1, b = 1,
γ = 1, Wiener
L = 512, σH = 0.5,
σP = 0.1, Wiener
L = 256, γ = 0.5, c = 2,
I = 10, J = 10, Wiener

Table 2. Parameter sets that optimize the criteria.
Parameter set that maxiParameter set that maximizes average SDR
mizes SDR of P
L = 1024, γ = 1,
κ = 0.25, None
L = 1024, γ = 1,
κ = 1.12, None
L = 512, a = 2, b = 4,
γ = 0.5, None
L = 1024, σH = 0.1,
σP = 0.5, Wiener
L = 512, γ = 0.5, c = 0.6,
I = 10, J = 3, Wiener

strictly satisﬁed. However, it is also possible to set the parameter
other than 0.5, such as 0.3, to suppress the effects from the verystrong components.
When ξ = 2γ, assuming that κ ≈ 1, the objective function canbe minimized by iterating the following formulae [5],Hτ,k 2γ ← ατ,k Wτ,k 2γ /(ατ,k + βτ,k ),Pτ,k 2γ ← βτ,k Wτ,k 2γ /(ατ,k + βτ,k ),where ατ,k = (Hτ +1,k γ + Hτ −1,k γ )2 ,βτ,k = κ2 (Pτ,k+1 γ + Pτ,k−1 γ )2 .

(12)
(13)
(14)
(15)

Hereafter, we refer to this algorithm as “HM2.” If we set ξ = 0.5, themethod is advantageous because the constraint h(t) + p(t) = w(t)is strictly satisﬁed. Besides, the dynamic range of the power spectrogram is sufﬁciently suppressed, because the value of γ is 0.25,
which is sufﬁciently small.
3.2.2. Simultaneous Optimization of Continuity and Error Criteria
Another way to realize the concept (B) is to add a criterion to theobjective function (Eq. 5 and Eq. 6), that measures the error between
the sum H + P and the original spectrogram W ,The objective function Eq. 5 is based on the square error of theboth spectrograms. This function can be minimized by iterating thefollowing formulae, and we refer to this algorithm as “SE,”(Ht+1,k γ + Ht−1,k γ ) + b(Wt,k γ − Pt,k γ ), (16)2+bγγγγa(Pt,k+1 + Pt,k−1 ) + b(Wt,k − Ht,k )Pt,k γ ←. (17)
2a + bHt,k γ ←The objective function Eq. 6 is based on I-divergence (KullbackLeibler-divergence), which is known as an effective criterion tomeasure the divergence between two spectrogrmas. In this case, γshould be 0.5 for scale invariance, and the updating formulae can be
written as follows [2, 8]:
Hτ,k
Pτ,k
mτ,k

←

{B1 + (B12 + 4A1 C1 )1/2 /2A1 }2

(18)

←
←

{B2 + (B22 + 4A2 C2 )1/2 /2A2 }2

(19)
(20)

Hτ,k /(Hτ,k + Pτ,k ),2
2+ 2, A2 = 2/σP
+ 2,
where A1 = 2/σH

(21)

2
B1 = (Hτ +1,k 1/2 + Hτ −1,k 1/2 )/σH
,

(22)

2
B2 = (Pτ,k+1 1/2 + Pτ,k−1 1/2 )/σP,
C1 = 2mτ,k Wτ,k , C2 = 2(1 − mτ,k )Wτ,k .

(23)
(24)

We refer to this algorithm as “Idiv.”467

L = 1024, γ = 0.5,
κ = 1.2, Wiener
L = 1024, γ = 1,
κ = 1, Wiener
L = 512, a = 4, b = 2,
γ = 1, Wiener
L = 1024, σH = 0.1,
σP = 0.5, Wiener
L = 256, γ = 0.5, c = 4,
I = 10, J = 10, Wiener

Parameter set that maximizes average SIR
L = 1024, γ = 0.3,
κ = 1, Binary
L = 1024, γ = 1,
κ = 1.12, Binary
L = 512, a = 1.2, b = 2,
γ = 0.3, Binary
L = 1024, σH = 0.1,
σP = 0.5, Binary
L = 1024, γ = 0.5, c =0.25, I = 20, J = 3, Binary

4. PERFORMANCE COMPARISON OF HPSSALGORITHMS
In this section, we ﬁrst tune the parameters of each HPSS algorithm
using real music signals, and then evaluate each algorithm using
other music signals. The performance criteria we used were SDR(signal to distortion ratio) of H , SDR of P , average SDR of H and
P , and average SIR (signal to interference ratio) [9]. High SDR
indicates that separated signals have low distortion, and high SIR indicates that there remain few opposite components (e.g., percussive
component in H ).
4.1. Parameter Tuning
Table 1 shows the list of parameters to be determined in this experiment. Because there are many parameters and possible parameter values, there are too many parameter combinations. Therefore,
we ﬁrst selected the candidate parameters, and reduced the permutations to be tested, based on the techniques of design of experiments(DOE) [10]. The data set we used for parameter tuning was the same
one to the one which was used in [1]. Part of this data set was extracted from MASS database [11]. The number of music signals was
6, and the genre of each piece was different each other. All clips
were 16 kHz-sampled monaural signals, and the duration of eachclip was around 10–20 [s].
Table 2 shows the parameter values that optimize the evaluation
criteria. Some of parameters to optimize SDR of H and SDR of
P tend to be symmetry, e.g., as to (σH , σP ), (0.1, 0.5) is optimalfor SDR of H , while (0.5, 0.1) is optimal for SDR of P . Similar
symmetric properties are observed in κ in HM2. The parameter setsthat makes the average SDR and SIR optimal were quite similar tothe one that optimize the SDR of P , possibly because of the biased
power ratio of harmonic components to percussive components in anormal music signal.
As to time-frequency masking, the result shows that Wiener
mask is the best to improve SDR in many cases, while binary mask
is more preferable to suppress the volume of interference signal.
4.2. Performance Evaluation
We evaluated the performance of each method using the parameterswhich were decided in the previous section. We examined the performance of the methods using other 9 mixtures, which were used in
SiSEC evaluations1 .
Fig. 3 shows the improvement of SDR and SIR of the signals
separated by each method for each mixture. The result shows that
1 http://sisec.wiki.irisa.fr

1.5

5. SUMMARYSDR improvement [dB]1

In this paper, we described various harmonic/percussive sound separation algorithms, which were based on anisotropic continuity ofspectrogram: harmonic component is horizontally continuous andpercussive component is vertically continuous. There are many ways
to formulate HPSS algorithms based on the concept, and each algorithm has its own parameters that need to be tuned. In this paper
we investigated the parameters sets of each HPSS algorithm to maximize several types of criteria, using real music signals. Further, we
conducted comparative evaluations of the HPSS algorithms.
According to the experiments, we veriﬁed the inclination of parameter sets that maximize the performance of each method. Comparative evaluation based on the determined parameter sets suggeststhat the algorithm “HM2” and “Idiv” are comparatively better, while
they are not necessarily best to improve SDR of harmonic component, i.e., it should depend on the demands from the application. Our
future work will include the investigation on the compatibility to applications e.g. chord estimation.

0.5
0
-0.5
-1
-1.5
HM1

HM2

Idiv
Method

SE

2DF

SE

2DF

(a)
8
SDR improvement [dB]

6
4
2
0
-2
-4
-6
-8
HM1

HM2

Idiv
Method

6. REFERENCES(b)
7
SDR improvement [dB]

6
5
4
3
2
1
0
-1
-2
HM1

HM2

Idiv
Method

SE

2DF

SE

2DF

(c)

SIR improvement [dB]

25
20
15
10
5
0
HM1

HM2

Idiv
Method(d)Fig. 3. (a) SDR improvement of H components for 9 songs, (b)SDR improvement of P components for the 9 songs, (c) average
SDR improvement of H and P for the 9 songs, (d) average SIR
improvement of H and P for the 9 songs.
the performance principally depends on the piece, but there are following tendencies: the performance of “2DF” is not as high as othermethods totally. There are no algorithms that maximize all the criteria. For example, if we prioritize the SDR of H over other criteria,
the algorithm that performs best is “SE” (Fig. 3 (a) ). However, theperformance of “SE” is lower than other three methods if we focused on the SDR of P . If we focused on the average SDR and SIR,
“HM2” and “Idiv” outperform other methods as a whole, indicating
the comparable performance of each other.

468[1] F. Rigaud, M. Lagrange, A. Röbel, and G Peeters, “Drum extraction from polyphonic music based on a spectro-temporalmodel of percussive sounds,” in Proc. ICASSP, 2011, pp. 381–
384.
[2] N. Ono, K. Miyamoto, H. Kameoka, J. Le Roux, Y. Uchiyama,
E. Tsunoo, T. Nishomoto, and S. Sagayama, “Harmonic andpercussive sound separation and its application to mir-related
tasks,” in Springer 274, 2010, pp. 213–236.
[3] J. Reed, Y. Ueda, S. M. Siniscalchi, Y. Uchiyama, and
S. Sagayama, “Minimum classiﬁcation error training to improve isolated chord recognition,” in proc. ICASSP, 2009, pp.
609–614.
[4] E. Tsunoo, N. Ono, and S. Sagayama, “Rhythm map: Extraction of unit rhythmic patterns and analysis of rhythmic structure from music acoustic signals,” in Proc. ICASSP, 2009, pp.
185–188.
[5] H. Tachibana, T. Ono, N. Ono, and S. Sagayama, “Melody
line estimation in homophonic music audio signals based on
temporal-variability of melodic source,” in Proc. ICASSP,
2010, pp. 425–428.
[6] C. L. Hsu, D. L. Wang, and J.-S. R. Jang, “A trend estimation
algorithm for singing pitch detection in musical recordings,” inProc. ICASSP, 2011, pp. 393–396.
[7] N. Ono, K. Miyamoto, J. Le Roux, H. Kameoka, and
S. Sagayama, “Separation of a monaural audio signal into harmonic/percussive components by complementary diffusion on
spectrogram,” in Proc. EUSIPCO, 2008.
[8] N. Ono, K. Miyamoto, H. Kameoka, and S. Sagayama, “A
real-time equalizer of harmonic and percussive components inmusic signals,” in Proc. ISMIR, 2008, pp. 139–144.
[9] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement in blind audio source separation,” in IEEE Transactions on Speech and Audio Processing, 2006, vol. 14, pp.
1462–1469.
[10] G. Taguchi, Design of Experiments, Maruzen, 1976.
[11] M. Vinyes, “MTG MASS database,” 2008, http://www.
mtg.upf.edu/static/mass/resources.