HARMONIC-TEMPORAL-TIMBRAL CLUSTERING (HTTC)FOR THE ANALYSIS OF MULTI-INSTRUMENT POLYPHONIC MUSIC SIGNALS
Kenichi Miyamoto, Hirokazu Kameoka† , Takuya Nishimoto, Nobutaka Ono and Shigeki Sagayama
Graduate School of Information Science and Technology
The University of Tokyo
Hongo, Bunkyo-ku, Tokyo, 113-8656, Japan{miyamoto, kameoka, nishi, onono, sagayama}@hil.t.u-tokyo.ac.jp
ABSTRACT

such as [3, 4], HTC deals with the structures in both time andfrequency directions simultaneously and shows high perforIn this paper, we discuss a new approach named Harmonicmance. In this paper, we present a new approach for multiTemporal-Timbral Clustering (HTTC) for the analysis of singleinstrument music analysis which simultaneously realizes achannel audio signal of multi-instrument polyphonic music toclustering of the spectral energy into acoustic events and aestimate the pitch, onset timing, power and duration of all theclassi¿cation of each acoustic event into timbre categories acacoustic events and to classify them into timbre categories sicording to their similarity with regard to the timbre feature.
multaneously. Each acoustic event is modeled by a harmonic
This approach is developed as an extension of HTC.
structure and a smooth envelope both represented by Gaussian mixtures. Based on the similarity between these spectroAlthough instrument recognition, modeling of musical intemporal structures, timbres are clustered to form timbre catstrument sounds and multipitch analysis have been consideredegories. The entire process is mathematically formulated as a
as dif¿cult problems, some approaches dealing with polyminimization problem for the I-divergence between the HTTC phonic signal have recently been developed [6]. These methparametric model and the observed spectrogram of the muods usually handle the problem of instrument information analsic audio signal to simultaneously update harmonic, tempoysis separately from multipitch analysis, resulting in a a tworal and timbral model parameters through the EM algorithm.
step approach which ¿rst extracts the features for each acousSome experimental results are presented to discuss the perfortic event from the result of multipitch analysis, conducted asmance of the algorithm.
a prior processing, then performs classi¿cation in the timbre
space [7, 8], training of the instrument sounds model [9] orIndex Terms— analysis of multi-instrument music, EMidenti¿cation of the source instrument through matching withalgorithm, Harmonic-Temporal-Timbral Clustering (HTTC)
learned timbre features [10, 11]. In another approach, a system visualizing frame-by-frame the probability of existence
1. INTRODUCTIONof each instrument frame-by-frame without estimating the F0
Analysis of single channel multi-instrument music signal has
information has also been developed [12].
been one of the ultimate goals of music signal processing,
However, we consider that multipitch analysis and timbre
with the ambition of obtaining a total estimation of each acousclustering
should be realized simultaneously. From the estic event, including the information on the instrument or thetimationofthe F0 frequency of polyphonic music, one cantimbre. This problem has a wide range of potential appliderive a timbre clustering of the acoustic events. Conversely,
cations including music transcription with part division, part
estimation of the timbre structure for each acoustic event cantracking, and music information retrieval (MIR). However, it
give clues for the separation of overlapped harmonics fromhas also been one of the most intricate problems, composedseveral events and reduce errors such as half-pitch or doubleof several dif¿cult sub-problems such as multipitch analysis
pitch errors, thus enabling higher performance of the multipand instrument recognition.
itch analysis.
So far, motivated by the psychological theory of auditory
scene analysis [1], we have developed a method for multiFrom another standpoint, when we listen to music perpitch analysis called Harmonic-Temporal Clustering (HTC)
formed with multiple instruments, we can usually naturally[2]. HTC decomposes the spectral energy of the signal inclassify acoustic events into timbre categories according to
the time-frequency domain into acoustic events, which aresimilarity of their timbre feature, even if the music is playedmodeled using acoustic object models with a harmonic andwith unknown instruments. In this paper, our goal is to protemporal 2-dimensional structure. It is thus able to estimate
pose a computational method which realizes a learning sysinformation such as F0 frequency, onset timing, etc., for eachtem similar to what humans perform.
acoustic event. Unlike conventional frame-wise approaches
We name the uni¿ed analysis we propose for multi-instrument† Now with NTT Communication Science Laboratories
music Harmonic-Temporal-Timbral Clustering (HTTC).

1-4244-1484-9/08/$25.00 ©2008 IEEE113

ICASSP 2008

Fig. 2. Example of Timbre Structure. left: piano, right: violin
less of the pitch, power, onset timing and duration.
We shall note that this de¿nition of timbre is simple anddoes not aim at expressing all the features of timbre in a singleinstrument. Timbre clustering in HTTC might thus face some
limitations as an instrument recognition method, and makesome mistakes as we humans sometimes do too.

Fig. 1. Generative Model in HTTC
2. APPROACH FOR HTTC

2.3. Justi¿cation of our approach

2.1. Generative Model for the Spectral Energy
Consider the observed power spectrum time series W (x, t)of a music acoustic signal, where x is log-frequency and t
is time. This spectrum W (x, t) is assumed to be generated
as the sum of the spectral energy corresponding to acoustic source events performed with different onset time, pitch,
power and duration and which belong to an unknown timbre
category, as is shown in Fig.1.
Therefore, the problem we have to solve is an inverse
problem to estimate the parameter set Θ which best approximates W (x, t) as the sum of K parametric acoustic object
models qk (x, t; Θ) corresponding to spectral energy patterns
originated from a single source. The parameter set Θ includes all the parameters of each acoustic event, such as F0
frequency, onset timing and the timbre category the source belongs to, and the parameters of each timbre feature, which weshall de¿ne in the following section. Estimation is performed
simultaneously on all the parameters.
2.2. De¿nition of Timbre
Although many de¿nitions of timbre or instrument have been
considered in previous works, it is still ambiguous. Some definitions treating three elements of music sounds, i.e., loudness, pitch and timbre, say that timbre is the total feature
which does not depend on F0 frequency and power. In addition, timbre feature can also be considered independent fromthe duration according to our general knowledge.
In this paper, timbre feature can be de¿ned as the shape ofspectral energy in time and log-frequency space which does
not depend on F0 , spectral power, onset, and duration. For
example, spectral energy shapes for the piano and the violin
are shown in Fig.2. There are many differences between these
two shapes, both in the spectral power of the partials and thetemporal structure. Therefore we can consider that the difference in timbre is derived from the difference of shape of thespectral energy, and that the shapes of acoustic events classi¿ed into the same timbre category should look alike regard-114

A very interesting point in music is that acoustic events fromthe same instrument but with different durations are recognized as belonging to the same timbre category. This showsthat a simple pattern clustering in the time-frequency domain
can not account for timbre, and that the clustering of the timbre categories and the estimation of the audio object parameters (including duration) need to be performed simultaneously. This justi¿es our approach to perform jointly the clustering of the timbre categories and the estimation of the audio
object parameters.
3. HTTC MODEL3.1. Parametric Model for the Timbre StructureFirst, as we de¿ned in the previous section, the timbre feature
is expressed in time-frequency space and can be modeled as aparametric structure corresponding to the energy distribution
originated from a single source. Assuming that we are dealing with pitched instruments in this paper, the source energy
is characterized by its harmonic structure and its time duration. We model this structure with a 2-dimensional Gaussian
Mixture Model (GMM) distribution Tc (x, t; Θ) described as:Tc (x, t; Θ) =Tc,n,y (x, t; Θ)(1)n,yn)2(t−yφ)2vc,n uc,n,y − (x−log− 2φ2
2σ 2
Tc,n,y (x, t; Θ) =
e
2πσφ


∀c,vc,n = 1, ∀c, ∀n,uc,n,y = 1,n

(2)
(3)

ywhere c denotes the index of the timbre category, n is theindex of the partial, y is the index of the GMM for time duration, vc,n and uc,y denote variables corresponding to therelative energy of the partials and the time envelope respectively in the c-th timbre structure, and φ and ψ denote the frequency and time spreads of every 2-dimensional Gaussian respectively, which are considered constant in the HTTC model.
The model’s shape is shown in Fig. 3,

objects at
mk (x, t) which decompose W (x, t) into acoustic
each coordinate (x, t) (0 ≤ mk (x, t) ≤ 1, k mk (x, t) =1), and optimizing the model parameters and the mask functions iteratively as proposed in our previous work [2]. Since
the partitioned cluster mk (x, t)W (x, t) is expected to be thespectral energy distribution of a single acoustic source, thisdecomposion of energy is bene¿cial not only for the estimation of the model parameters depending on each acoustic source but also for timbre clustering and estimation of thetimbre structure which needs the shape of a single acousticsource.

Fig. 3. Timbre Structure. left: the entire shape, upper right:
GMM in the log-frequency direction, lower right: GMM for the time
duration

4.2. Minimization of the Objective Function Using the EM
AlgorithmTable 1. Parameters of the HTTC Model

denotation
wkμkτkγkckvc,nuc,yIn order to estimate the model parameters, we introduce as objective function the sum of the distances, measured using theI-divergence, between the partitioned clusters mk (x, t)W (x, t)
and the acoustic source models qk (x, t; Θ), including the timbre structure Tck (x, t; Θ):J =Jk(5)physical meanings
total energy of k-th source
pitch contour of k-th source
onset time of k-th source
duration of k-th source
timbre category which k-th source belongs torelative energy of n-th harmonic in c-th timbre
coef¿cient of the power envelope of c-th timbrekJk

3.2. Acoustic Object Model using Timbre Structure
Next we describe the acoustic object model corresponding to
the energy of a single acoustic source. The 2-dimensional
shape of each object model can be expressed with the timbre
structure Tc (x, t; Θ) modeled in the previous section. Therefore, according to the de¿nition of timbre, in addition to thetimbre category ck deciding the shape of the object model, thek-th acoustic object has the pitch contour μk , the onset timeτk , the total energy wk , and the duration γk as parameterswhich are independent from the timbre structure.
Finally, in order to control the time duration of each object
independently of the time envelope of the timbre structure, themodel can multiply the parameters uck ,y of the time envelope
by a time muting function R(t; γk ) which is equal to 1 before
the duration γk and then drops to 0 as the sigmoid function
1.
1+ep(t−γk )Altogether, the k-th acoustic object model can be describedas
1vck ,n uck ,yqk (x, t; Θ) = wkp(yφ−γk )2πσφ1+en,y−e(x−μk −log n)2(t−τk −yφ)−2σ 22φ2.

=mk (x, t)W (x, t)
qk (x, t; Θ)(6)−(mk (x, t)W (x, t) − qk (x, t; Θ)dxdt.
Dmk (x, t)W (x, t) log

Our problem can be regarded as the minimization of (5).
The minimization of J can be realized by the iteration of
the update rule shown in Fig.4 and described below:(1) Update the mask functions mk (x, t)(2) Estimate the parameters depending on each acoustic
object such as wk , μk , τk , and γk(3) Estimate the timbre category ck of each object from Jk
as the discriminant function(4) Estimate the GMM parameters vc,n , uc,y representing
the timbre structure.
Since each step of this update rule can reduce the objectivefunction (5) successfully, the iteration of these update steps
can yield to locally optimal parameters. For length purposes,
we skip here the description of the details of the update equations for each parameters, which can be obtained analyticallyby the combination of an undetermined multipliers Lagrange’s
method and a k-means-like algorithm for the update of ck .
5. EXPERIMENTS AND DISCUSSION(4)

5.1. Experimental Setup
We experimentally tested our approach, HTTC, from an audio
input. An input waveform data of about 10 seconds in a violin sonata by Frank performed with violin and piano was obtained by converting the MIDI data into WAV data sampled at16kHz. In addition, the initial pitches and onset times of eachacoustic source model was set correctly in order this time todiscuss the performance of timbre clustering and learned timbre shape speci¿cally. The input piano-roll data and the converted wavelet spectral energy are shown in Fig.5.

4. UPDATING PARAMETERS
4.1. Decomposition of Spectral Energy
As we explained in the previous section, the problem we haveto solve is to estimate the model parameters Θ of each acoustic source and each timbre structure listed in Table 1 from theobserved energy pattern W (x, t). It can be solved throughan EM-like algorithm, introducing spectral masking functions

115

time

ytime

Lo
gfre
quenc

Minimize
Objective function JTimbre2

Timbre1ywk , μ k , τ k , γ kLogfrequenc

mk ( x, t )Log-frequency

Update parameters ofeach object,

Update mask functions

time

Update parameters
for timbre shape,

Timbre Clusteringwith J k fordiscriminant function

vc ,n , uc , yFig. 6. Experimental result with 2 timbre categories

violinLog-frequency

time

Timbre1

Timbre2

Timbre3

Fig. 5. Input Data. left: input piano-roll data, right: obtainedspectral energy distribution

time

time

time

Lo
gfre
quen
cy

time

Lo
gfre
quen
cy

time

Lo
gfre
quen
cy

Log-frequency

pianoLog-frequency

Fig. 4. 4-step Algorithm Minimizing the Objective Function

5.2. Experimental Results and Discussion
5.2.1. Result with the number of timbre categories set to 2

Fig. 7. Experimental result with 3 timbre categories

First, we tested our algorithm with the number of timbre set
to 2. The obtained piano-roll data and timbre shape are shownin Fig.6. We can see in the ¿gure that many notes of the violin
and some notes of the piano in the low frequencies were classi¿ed into the same timbre category. We think that the reason
for this is that the harmonic structures are different between
notes with different F0 within a single instrument.

7. REFERENCES

5.2.2. Result with the number of timbre categories set to 3
We tested next the same input signal with the number of timbre categories set to 3. The obtained piano-roll data and timbre shape are shown in Fig.7. In this result, most of the events
belonging to the violin were classi¿ed into the third category(blue line) and few notes of the piano in the low frequencies
were classi¿ed into it. Therefore, we think that notes of theviolin and the piano were well classi¿ed into different timbre
categories.
6. CONCLUSION AND FUTURE WORKIn this paper, we proposed a new approach for multi-instrument
musical analysis, called Harmonic-Temporal-Timbral Clustering (HTTC), and we experimentally evaluated the performance of HTTC with a music signal consisting of piano andviolin. Future work will include the design of a timbre structure model depending on F0 for a better instrument recognition.
This research was partly supported by MEXT Grant-inAid #17300054.

116[1] A. S. Bregman, Auditory Scene Analysis, MIT Press, Cambridge,
1990.
[2] H. Kameoka et al. , “A Multipitch Analyzer Based on Harmonic Temporal Structured Clustering,” IEEE Trans. on Audio, Speech and Language Processing, in Press.
[3] A. Klapuri, “Multiple fundamental frequency estimation based on harmonicity and spectral smoothness,” IEEE Trans. Speech and Audio
Proc., 11(6), 804-816, 2003.
[4] M.Goto, “A Robust Predominant-F0 Estimation Method for Real-time
Detection of Melody and and bass lines in CD recordings,” in Proc.
ICASSP,2000.
[5] S. Godsill and M.Davy, “Bayesian Harmonic Models for Musical
Pitch Estimation and Analysis,” Proc. ICASSP2002, Vol. 2, pp. 17691772, 2002.
[6] T. Kitahara, “Computational Musical Instrument Recognition andIts Application to Content-based Music Information Retrieval,”
Ph. D. Thesis, Kyoto University, 2007.
[7] T. Kitahara, et al. ,“Musical Instrument Identi¿cation based on F0 dependent Multivariate Normal Distribution,” Proc. ICASSP2003,
Vol. 5, pp421-424, 2003.
[8] T. Kitahara, et al. ,“Category-level Identi¿cation of Non-registeredMusical Instrument Sounds,” Proc. ICASSP2004, Vol. 4, pp253-256,2004.
[9] A. Klapuri, “Analysis of Musical Instrument Sounds by Source-FilterDecay Model,” Proc, ICASSP, 2007.
[10] J. Eggink, G. J. Brown, “A Missing Feature Approach to Instrument
Identi¿cation in Polyphonic Music,” Proc, ICASSP, 2003.
[11] K. Kashino, Hiroshi Murase,“A sound source identi¿cation system forensemble music based on template adaptation and music stream extraction,” Speech Communication,vol.27, 1999.
[12] T. Kitahara, M. Goto, K. Komatani, T. Ogata, and H. G. Okuno, “Instrogram: Probabilistic Representation of Instrument Existence for
Polyphonic Music”, IPSJ Journal, Vol. 48, No. 1, pp. 214–226, 2007.