INSTROGRAM: A NEW MUSICAL INSTRUMENT RECOGNITION TECHNIQUEWITHOUT USING ONSET DETECTION NOR F0 ESTIMATION
Tetsuro Kitahara,† Masataka Goto,‡ Kazunori Komatani,† Tetsuya Ogata† and Hiroshi G. Okuno††
Dept. of Intelligence Science and Technology
Graduate School of Informatics, Kyoto University
Sakyo-ku, Kyoto 606-8501, Japan{kitahara, komatani, ogata, okuno}@kuis.kyoto-u.ac.jp

ABSTRACTThis paper describes a new technique for recognizing musicalinstruments in polyphonic music. Because the conventional framework for musical instrument recognition in polyphonic music had
to estimate the onset time and fundamental frequency (F0) of eachnote, instrument recognition strictly suffered from errors of onset
detection and F0 estimation. Unlike such a note-based processing framework, our technique calculates the temporal trajectory of
instrument existence probabilities for every possible F0, and theresults are visualized with a spectrogram-like graphical representation called instrogram. The instrument existence probability isdefined as the product of a nonspecific instrument existence probability calculated using PreFEst and a conditional instrument existence probability calculated using the hidden Markov model. Experimental results show that the obtained instrograms reflect theactual instrumentations and facilitate instrument recognition.
1. INTRODUCTION
Musical instrument recognition is an important task for many applications including automatic music transcription, music information retrieval and computational auditory scene analysis. In particular, recent worldwide popularization of online music distribution
services and portable digital music players makes musical instrument recognition more important. This is because musical pieces,
especially classical music, are characterized by what instruments
are used. In fact, the names of some music genres are based on instrument names, such as “piano sonata” and “string quartet.” Musical instrument recognition can therefore be used when one wants
to search for certain types of musical pieces, such as piano sonata
or string quartet.
Whereas musical instrument recognition studies mainly dealt
with solo musical sounds in 1990s (e.g., [1]), the number of thosedealing with polyphonic music has been increasing in recent years.
Kashino et al. [2] developed a computational music scene analysis
architecture called OPTIMA, which recognizes musical notes andthe instruments based on the Bayesian probability network. They
subsequently proposed a method that identifies the instrument
playing each musical note based on template matching with template adaptation [3]. Kinoshita et al. [4] improved the robustnessof OPTIMA to the overlapping of frequency components, which
occurs when multiple instruments play simultaneously, based onfeature adaptation. Eggink et al. [5] tackled this overlapping problem with the missing feature theory. They subsequently dealt withthe problem of identifying only the instrument playing the main
This research was partially supported by the Ministry of Education,
Culture, Sports, Science and Technology (MEXT), Grant-in-Aid for Scientific Research (A), No.15200015, and Informatics Research Center for Development of Knowledge Society Infrastructure (COE program of MEXT,
Japan).

1­4244­0469­X/06/$20.00 ©2006 IEEE‡

National Institute of Advanced Industrial
Science and Technology (AIST)
Tsukuba, Ibaraki 305-8568, Japanm.goto@aist.go.jp(the most predominant) melody on the assumption that the main
melody’s partials suffer less from other sounds occurring simultaneously [6]. Vincent et al. [7] proposed a new musical instrument
identification method based on independent subspace analysis. Kitahara et al. [8] proposed techniques for feature weighting based on
the robustness to the above-mentioned overlapping problem andfor avoiding musically unnatural errors using musical context.
The common feature in the above-mentioned studies except
for [7] is that instrument identification is performed for each frameor each note. In the former case [5, 6], it is difficult to obtain a reasonable accuracy because temporal variations of spectra are important characteristics of musical instrument sounds. In the lattercase [2, 3, 4, 8], the identification system has to first estimate theonset time and fundamental frequency (F0) of musical notes andthen extracts the harmonic structure of each note based on the estimated onset time and F0. Therefore, instrument identification
suffers from errors of onset detection and F0 estimation. In the experiments reported in [3] and [8], in fact, correct data of the onset
times and F0s were manually fed.
In this paper, we propose a new technique that recognizes musical instruments in polyphonic musical audio signals without relying on onset detection nor F0 estimation. The key idea of ourtechnique is to visualize the probability that the sound of each oftarget instruments exists at each time and each frequency. The result of this analysis is a set of images, called instrogram, that are
similar to a spectrogram except in that each point represents not
the energy of the signal but the instrument existence probability.
This probability is defined as the product of two kinds of probabilities, called nonspecific instrument existence probability and conditional instrument existence probability, and these are calculatedusing PreFEst [9] and the hidden Markov model (HMM), respectively. The advantage of our technique is that errors of calculating
one probability do not influence the calculation of the other because the two probabilities can be calculated independently.
2. INSTROGRAM
The instrogram is a spectrogram-like graphical representation ofa musical audio signal, which is useful for finding which instruments are used in the signal. There exists one image for each target instrument, and each image, where the horizontal and verticalaxes represent time and frequency respectively, shows the probability that the target instrument is used. An example is presented
in Fig. 1. This example is the result of analyzing an audio signalof “Auld Lang Syne” with respect to piano, violin, clarinet, andflute. This signal, played on piano, violin, and flute, was preparedwith the procedure described later, in Section 4. If the instrogramis too detailed for some purposes, it can be simplified by dividing the whole frequency region into some subregions and merging
results within each subregion. The simplified version of Fig. 1 isgiven in Fig. 2. From the four images of the instrogram or fromV ­ 229ICASSP 2006

Note Number

Note Number

Note Number

Note Number

Piano
80
70
60
50
0

5

10

15
Violin

20

25

30

0

5

10

15
Clarinet

20

25

30

0

5

10

15
Flute

20

25

30

0

5

10

15
Time [sec]

20

25

30

80
70
60
50
80
70
60
50
80
70
60
50

Fig. 1. An example of instrograms. This is the result of analyzing
trio music, “Auld Lang Syne,” played on piano, violin, and flute.

Instrument (Note Number)

PF (75−80)
PF (69−74)
PF (63−68)
PF (57−62)
PF (51−56)
PF (45−50)
VN (75−80)
VN (69−74)
VN (63−68)
VN (57−62)
VN (51−56)
VN (45−50)
CL (75−80)
CL (69−74)
CL (63−68)
CL (57−62)
CL (51−56)
CL (45−50)
FL (75−80)
FL (69−74)
FL (63−68)
FL (57−62)
FL (51−56)
FL (45−50)
0

5

10

15
Time [sec]

20

25

30

Fig. 2. The simplified (summarized) instrogram of Fig. 1.
the simplified instrogram, we can see that the melodies in the high(approx. note numbers 70–80), middle (60–75), and low (45–60)
pitch regions are played on flute, violin, and piano.

p(exist; t, f ) as this weight because this weight can be interpreted
as the probability that there exists a certain sound at its F0.
The conditional instrument existence probability, on the other
hand, is calculated by using HMMs because temporal characteristics of an instrument sound are important in recognizing its instrument. In each possible frequency f , the temporal trajectory
H(t, f ) of the harmonic structure with F0 of f can be considered
to be generated from a Markov chain of m + 1 models of possible
instruments ω1 , · · · , ωm and silence. Each model is an HMM thatconsists of multiple states. Then, p(ωi |exist; t, f ) can be calculated from the likelihoods of paths in the chain.
The instrument existence probability p(ωi ; t, f ) can be estimated robustly because the two constituent probabilities corresponding to the note estimation and instrument identification arecalculated independently and then integrated by multiplying them.
In most previous studies, the onset time and F0 of each note werefirst estimated and then the instrument of the note was identified
by analyzing spectral components extracted based on the result of
the note estimation. The upper limit of the instrument identification performance was therefore bound by the precedent note estimation, which is generally difficult and not robust for polyphonicmusic. Unlike such note-based symbolic approach, our technique
is based on a non-symbolic and non-sequential approach that ismore robust for polyphonic music.
3.1. Nonspecific Instrument Existence Probability
By using the PreFEst, the nonspecific instrument existence probability p(exist; t, f ) is estimated on the basis of the maximum likelihood estimation without assuming the number of sound sourcesin a mixture. The PreFEst, which was originally developed for estimating F0s of melody and bass lines, consists of three processes:
PreFEst-front-end for frequency analysis, PreFEst-core for estimating the relative dominance of every possible F0, and PreFEstback-end for evaluating the temporal continuity of the F0. Because
the problem to be solved here is not to estimate the predominant
F0s as melody and bass lines but to calculate p(exist; t, f ) of every
possible F0 f , we only use the PreFEst-core.
The PreFEst-core models an observed power spectrum as aweighted mixture of tone models p(x|F ) of every possible F0 F .
The tone model p(x|F ), where x is the log frequency, represents a
typical spectrum of the harmonic structure, and the mixture densityp(x; θ(t) ) is defined asp(x; θ(t) ) = Fhw(t) (F )p(x|F )dF,

Flθ(t) = {w(t) (F )|Fl ≤ F ≤ Fh},

3. ALGORITHM FOR CALCULATING INSTROGRAM
Each image of the instrogram is a plane with horizontal and vertical axes representing time and frequency. The intensity of thecolor of each point (t, f ) in the image represents the probabilityp(ωi ; t, f ) that a sound of the target instrument ωi exists at time tand frequency f . We call this instrument existence probability. It
can be calculated as the product of two probabilities:p(ωi ; t, f ) = p(exist; t, f ) p(ωi |exist; t, f ),where p(exist; t, f ) called nonspecific instrument existence probability is the probability that a sound of a certain instrument exists attime t and frequency f , while p(ωi |exist; t, f ) called conditional
instrument existence probability is the conditional probability that,
if a sound of a certain instrument exists at time t and frequency f ,the instrument is ωi .
The nonspecific instrument existence probability is calculatedusing PreFEst [9]. PreFEst models, at each frame, an observed
spectrum of the input signal containing multiple musical instrument sounds as a weighted mixture of harmonic-structure tonemodels with every possible F0. The weight of each tone model
represents how relatively predominant its tone model is. We define

where Fl and Fh denote the lower and upper limits of the possible F0 range and w(t) (F ) is the weight of a tone model p(x|F ) Fhthat satisfies Fl w(t) (F )dF = 1. If we can estimate the model
parameter θ(t) such that the observed spectrum is likely to havebeen generated from p(x; θ(t) ), the spectrum can be considered to
be decomposed into harmonic-structure tone models and w(t) (F )can be interpreted as the relative predominance of the tone modelwith F0 of F at time t. We therefore define the nonspecific instrument existence probability p(exist; t, f ) to be equal to w(t) (f ).
The weights can be estimated using the Expectation-Maximization(EM) algorithm as described in [9].
3.2. Conditional Instrument Existence Probability
The conditional instrument existence probability p(ωi |exist; t, f )is calculated through the following steps.
3.2.1. Short-time Fourier transform
The spectrogram of the given audio signal is calculated with theshort-time Fourier transform (STFT) shifted by 10 ms (441 points
at 44.1 kHz sampling) with an 8192-point Hamming window.

V ­ 230

Spectral centroid
Relative power of fundamental component
Relative cumulative power from fundamental to i-thcomponents (i = 2, 3, · · · , 9)11
Relative power in odd and even components
12 – 20 Number of components whose duration is p% longer
than the longest duration (p = 10, 20, · · · , 90)
Temporal features

detailed results. If the instrogram is used for retrieving musicalpieces including a certain instrument’s sounds, for example, instrument existence probabilities for rough frequency regions (e.g.,high, middle and low) are sufficient. We therefore divide the whole
frequency region into N subregions I1 , · · · , IN and calculate theinstrument existence probability p(ωi ; t, Ik ) for k-th frequency region Ik . p(ωi ; t, Ik ) is defined as p(ωi ; t, f ∈I f ), which can bekobtained by iteratively calculating the following equation because
the frequency axis is practically discrete.
p(ωi ; t, f1 ∪ · · · ∪ fi ∪ fi+1 )= p(ωi ; t, f1 ∪ · · · ∪ fi ) + p(ωi ; t, fi+1 )−p(ωi ; t, f1 ∪ · · · ∪ fi ) p(ωi ; t, fi+1 ),21

where Ik = {f1 , · · · , fi , fi+1 , · · · , fnk }.

Table 1. Overview of 28 features
Spectral features1
2
3 – 10

Gradient of straight line approximating powerenvelope
22 – 24 The temporal mean of differentials of power envelope from t to t + iT /3 (i = 1, · · · , 3)
Modulation features
25 , 26
27 , 28

Amplitude and Frequency of AM
Amplitude and Frequency of FM

3.2.2. Harmonic structure extraction
In each possible frequency f , the temporal trajectory H(t, f ) of
the harmonic structure (10 harmonics) whose F0 is f is extracted.
3.2.3. Feature extraction
Designing effective features is an important issue in musical instrument recognition. In the field of speech recognition studies,mel-frequency cepstrum coefficents (MFCCs) and Delta MFCCs
are commonly used. Although these features may be considered
possible to be applied to musical instrument recognition, we designed features optimized for musical instrument sounds becausemusical instrument sounds have more complicated temporal variations (e.g., amplitude and frequency modulations). We adopt 28
features listed in Table 1, which are partly modified from those
we previously proposed [8]. For every time t (every 10 ms in theimplementation), we first excerpt a T -length bit of the harmonicstructure trajectory Ht (τ, f ) (t ≤ τ < t + T ) from the whole
trajectory H(t, f ) and then extract a feature vector x(t, f ) consisting of 28 features from Ht (τ, f ). T is 50 ms in the current
implementation.
3.2.4. Probability calculation
We train HMMs, each consisting of 15 states, for target instruments ω1 , · · · , ωm and silence, and then consider the time seriesx(t, f ) of feature vectors to be generated from a Markov chain of
those m + 1 HMMs. The problem to be solved here is thus tocalculate, for each HMM Mi (i = 1, · · · , m, silence), the probability p(x(t, f )|Mi ; t) that the feature vector x(t, f ) is generated
from Mi at time t. Because the Markov chain assumes that themodel at time t only depends on that at the previous time t − 1,this probability can be calculated byp(x(t, f )|Mi ; t) =p(x(t, f )|Mi )p(Mi |Mj ),jwhere p(x(t, f )|Mi ) is the probability that the model Mi generates the feature vector x(t, f ) at any time and p(Mi |Mj ) is thetransition probability from Mj to Mi . Here p(x(t, f )|Mi ) is calculated by the trained HMM Mi . The conditional instrument existence probability p(ωi |exist; t, f ) can thus defined to be equal top(x(t, f )|Mi ; t).
3.3. Simplifying Instrograms
The instrogram calculates instrument existence probabilities forevery possible frequency, but some applications do not need such

3.4. Application to Musical Instrument Identification
The simplest way for applying instrograms to musical instrument
identification is to obtain the instrument maximizing p(ωi ; t, f )for each time t and frequency f . However, the results obtained inthis way include some errors because it does not take the temporal continuity into consideration. Here, we consider the time series of the instruments maximizing p(ωi ; t, f ) for each frequency
to be outputs of a Markov chain whose states are the instruments
ω1 , · · · , ωm and silence. In the Markov chain, the transition probabilities from a state to the same state, from a non-silence state to
the silence state, and from the silence state to a non-silence state
are more than zero and the other probabilities are zero. Once themost likely path in the chain is obtained, the times when the instrument ωi begins and stops playing can be estimated from the timesof transitions from the silence state to the ωi state and vice versa,
respectively.
4. EXPERIMENTS
We conducted experiments on obtaining instrograms from audio
signals. We used trio music of “Auld Lang Syne” that Kashino
et al. used [2]. Audio data of Auld Lang Syne were generated
by mixing audio data of RWC-MDB-I-2001 [10] on a computeraccording to a standard MIDI file that we input using a MIDI sequencer based on Kashino’s score. Target instruments were piano(PF), violin (VN), clarinet (CL), and flute (FL). The time resolution was 10 ms from the beginning to the end of the musical piece,
and the resolution was every 20 cent from A2 to A5. The width ofeach frequency region Ik was 600 cent. We used Fujihara’s implementation [11] for PreFEst and HTK 3.0 for HMMs.
The results of generating instrograms for four different instrumentations are shown in Fig. 3. Note that Fig. 3 shows only simplified instrograms due to limited space. Comparing (a) and (b),we can see that (a) has high existence probabilities for the flute inhigh frequency regions while (b) has very low (almost zero) probabilities. In addtion, (c) has the highest existence probabilities forthe violin and the secondarily highest existence probabilities forthe piano, whereas it has lower probabilities for the other instruments. Similarly, (d) has high existence probabilities for the piano,
whereas it has lower probabilities for the other instruments.
Next, we conducted experiments on musical instrument recognition using the obtained instrograms based on the method in Section 3.4. The accuracy of the recognition is evaluated with
(Num of correctly recognized frames)(Recognition rate) =.
(Num of the whole frames)
The results are listed in Table 2. From the table, we can see thatthe instrogram was useful for musical instrument recognition.
5. CONCLUSIONS
We have described a new instrogram representation obtained by a
new musical instrument recognition technique that does not relyV ­ 231

PF (75−80)
PF (69−74)
PF (63−68)
PF (57−62)
PF (51−56)
PF (45−50)

VN (75−80)
VN (69−74)
VN (63−68)
VN (57−62)
VN (51−56)
VN (45−50)

VN (75−80)
VN (69−74)
VN (63−68)
VN (57−62)
VN (51−56)
VN (45−50)

Instrument (Note Number)

Instrument (Note Number)

PF (75−80)
PF (69−74)
PF (63−68)
PF (57−62)
PF (51−56)
PF (45−50)

CL (75−80)
CL (69−74)
CL (63−68)
CL (57−62)
CL (51−56)
CL (45−50)
FL (75−80)
FL (69−74)
FL (63−68)
FL (57−62)
FL (51−56)
FL (45−50)

CL (75−80)
CL (69−74)
CL (63−68)
CL (57−62)
CL (51−56)
CL (45−50)
FL (75−80)
FL (69−74)
FL (63−68)
FL (57−62)
FL (51−56)
FL (45−50)

0

5

10

15
Time [sec]

20

25

30

0

5

15
Time [sec]

20

25

30

20

25

30

(b) VN–CL–PF

PF (75−80)
PF (69−74)
PF (63−68)
PF (57−62)
PF (51−56)
PF (45−50)

PF (75−80)
PF (69−74)
PF (63−68)
PF (57−62)
PF (51−56)
PF (45−50)

VN (75−80)
VN (69−74)
VN (63−68)
VN (57−62)
VN (51−56)
VN (45−50)

VN (75−80)
VN (69−74)
VN (63−68)
VN (57−62)
VN (51−56)
VN (45−50)

Instrument (Note Number)

Instrument (Note Number)(a) FL–CL–PF

10

CL (75−80)
CL (69−74)
CL (63−68)
CL (57−62)
CL (51−56)
CL (45−50)
FL (75−80)
FL (69−74)
FL (63−68)
FL (57−62)
FL (51−56)
FL (45−50)

CL (75−80)
CL (69−74)
CL (63−68)
CL (57−62)
CL (51−56)
CL (45−50)
FL (75−80)
FL (69−74)
FL (63−68)
FL (57−62)
FL (51−56)
FL (45−50)

0

5

10

15
Time [sec]

20

25

30

0

(c) VN–VN–PF

5

10

15
Time [sec](d) PF–PF–PF

Fig. 3. Instrograms of “Auld Lang Syne” with four different instrument combinations. “FL–CL–PF” means that the treble, middle, andbass parts are played on flute, clarinet, and piano, respectively.
[2] K. Kashino et al., “Application of the Bayesian ProbabilTable 2. Recognition rate for each instrument combination.
ity Network to Music Scene Analysis,” Computational Auditory Scene Analysis, D. F. Rosenthal and H. G. Okuno (eds.),
PF–PF–PF
97.1%
VN–CL–PF 73.2%
Lawrence Erlbaum Associates, pp.115–137, 1998.
PF–VN–PF 84.7%
FL–PF–PF
82.8%[3] K. Kashino and H. Murase, “A Sound Source Identification
PF–CL–PF
86.9%
FL–VN–PF 78.5%
System for Ensemble Music based on Template Adaptation
VN–PF–PF 81.1%
FL–CL–PF
78.7%
and Music Stream Extraction,” Speech Communication, 27,
VN–VN–PF 86.7%
pp.337–349, 1999.
on both onset detection and F0 estimation. Whereas most previous
studies first estimated the onset time and F0 of each note and thenidentified the instrument of each note, our technique calculates the
instrument existence probability for each target instrument in eachpoint of the time-frequency plane. This non-symbolic approach
made it possible to avoid bad influences caused by errors of theonset detection and F0 estimation. In addition, by introducing a
Markov chain whose states correspond to target instruments andsilence for every possible F0, we achieved the identification ofmusical instruments (i.e., to symbolize the instrogram analysis)for polyphonic music.
Although we have applied instrograms to musical instrument identification in this paper, the instrograms have a wider
range of potentials. If the similarity between the instrograms oftwo musical pieces can be calculated, for example, it will make
instrumentation-similarity-based music information retrieval possible. Future work will include the development of such useful
applications of instrograms as well as the improvement of the accuracy of calculating the instrument existence probabilities.
Acknowledgments: We thank everyone who has contributed tobuilding and distributing the RWC Music Database (RWC-MDBI-2001) [10]. We also thank Mr. Hiromasa Fujihara for giving us
permission to use his program.
6. REFERENCES[1] K. D. Martin, “Sound-Source Recognition: A Theory andComputational Model,” PhD Thesis, MIT, 1999.

[4] T. Kinoshita et al., “Musical Sound Source Identification
based on Frequency Component Adaptation,” Proc. IJCAICASA Workshop, pp.18–24, 1999.
[5] J. Eggink and G. J. Brown, “A Missing Feature Approach
to Instrument Identification in Polyphonic Music,” Proc.
ICASSP, V, pp.553–556, 2003.
[6] J. Eggink and G. J. Brown, “Instrument Recognition in
Accompanied Sonatas and Concertos,” Proc. ICASSP, IV,
pp.217–220, 2004.
[7] E. Vincent and X. Rodet, “Instrument Identification in Soloand Ensemble Music using Independent Subspace Analysis,”
Proc. ISMIR, pp.576–581, 2004.
[8] T. Kitahara et al., “Instrument Identification in PolyphonicMusic: Feature Weighting with Mixed Sounds, Pitchdependent Timbre Modeling, and Use of Musical Context,”
Proc. ISMIR, pp.558–563, 2005.
[9] M. Goto, “A Real-time Music-scene-description system:
Predominant-F0 Estimation for Detecting Melody and Bass
Lines in Real-world Audio Signals,” Speech Communication,
43, pp.311–329, 2004.
[10] M. Goto et al., “RWC Music Database: Music Genre
Database and Musical Instrument Sound Database,” Proc. ISMIR, pp.229–230, 2003.
[11] H. Fujihara et al., “Singer Identification based on Accompaniment Sound Reduction and Reliable Frame Selection,” Proc.
ISMIR, pp.329–336, 2005.

V ­ 232