Vol. 48

No. 1

IPSJ Journal

Jan. 2007

Regular Paper

Instrogram: Probabilistic Representation ofInstrument Existence for Polyphonic Music
Tetsuro Kitahara,† Masataka Goto,†† Kazunori Komatani,†
Tetsuya Ogata† and Hiroshi G. Okuno†
This paper presents a new technique for recognizing musical instruments in polyphonicmusic. Since conventional musical instrument recognition in polyphonic music is performed
notewise, i.e., for each note, accurate estimation of the onset time and fundamental frequency(F0) of each note is required. However, these estimations are generally not easy in polyphonicmusic, and thus estimation errors severely deteriorated the recognition performance. Without
these estimations, our technique calculates the temporal trajectory of instrument existence
probabilities for every possible F0. The instrument existence probability is deﬁned as the
product of a nonspecific instrument existence probability calculated using the PreFEst anda conditional instrument existence probability calculated using hidden Markov models. The
instrument existence probability is visualized as a spectrogram-like graphical representation
called the instrogram and is applied to MPEG-7 annotation and instrumentation-similaritybased music information retrieval. Experimental results from both synthesized music and real
performance recordings have shown that instrograms achieved MPEG-7 annotation (instrument identiﬁcation) with a precision rate of 87.5% for synthesized music and 69.4% for real
performances on average and that the instrumentation similarity measure reﬂected the actual
instrumentation better than an MFCC-based measure.

strument recognition studies mainly dealt withsolo musical sounds in the 1990s 1) , the number of studies dealing with polyphonic music
has been increasing in recent years. Kashino,
et al. 2) developed a computational architecture
for music scene analysis called OPTIMA, which
recognizes musical notes and instruments basedon the Bayesian probability network. They subsequently proposed a method that identiﬁes the
instrument playing each musical note based on
template matching with template adaptation 3) .
Kinoshita, et al. 4) improved the robustness of
OPTIMA to the overlapping of frequency components, which occurs when multiple instruments are played simultaneously, based on feature adaptation. Eggink, et al. 5) tackled thisoverlap problem with the missing feature theory. They subsequently dealt with the problem of identifying only the instrument playing the main melody based on the assumption that the main melody’s partials would suffer less from other sounds occurring simultaneously 6) . Vincent, et al. 7) formulated both music transcription and instrument identiﬁcation
as a single optimization based on independent
subspace analysis. Essid, et al. 8) achieved F0estimation-less instrument recognition based ona priori knowledge about the instrumentationfor ensembles. Kitahara, et al. 9) proposed techniques to solve the above-mentioned overlap

1. Introduction
The goal of our study is to enable users toretrieve musical pieces based on their instrumentation. The types of instruments that areused are important characteristics for retrieving musical pieces. In fact, the names of certain musical genres, such as “piano sonata”
and “string quartet”, are based on instrument names. There are two strategies for
instrumentation-based music information retrieval (MIR). The ﬁrst allows users to directly
specify musical instruments (e.g., searching fora piano solo or string music). This strategy isuseful because, unlike other musical elements
such as chord progressions, specifying instruments does not require any special knowledge.
The other strategy is Query-by-Example, whereonce users specify a musical piece that they like,the system searches for pieces that have similar instrumentation to the speciﬁed piece. This
strategy is also particularly useful for automatically generating playlists for background music.
The key technology for achieving the abovementioned MIR is to recognize musical instruments from audio signals. Although musical in† Department of Intelligence Science and Technology,
Graduate School of Informatics, Kyoto University†† National Institute of Advanced Industrial Scienceand Technology (AIST)214

Vol. 48

No. 1

Instrogram: Probabilistic Representation of Instrument Existence

problem and to avoid musically unnatural errors using musical context.
The common feature of these studies, except
for Refs. 7) and 8), is that instrument identiﬁcation is performed for each frame or eachnote. In the former case 5),6) , it is diﬃcult toobtain a reasonable accuracy because temporalvariations in spectra are important characteristics of musical instrument sounds. In the latter case 2)∼4),9) , the identiﬁcation system has toﬁrst estimate the onset time and fundamental
frequency (F0) of musical notes and then extract the harmonic structure of each note basedon the estimated onset time and F0. Therefore,
the instrument identiﬁcation suﬀers from errorsof onset detection and F0 estimation. In fact,correct data for the onset times and F0s were introduced manually in the experiments reported
in Refs. 3) and 9).
In this paper, we propose a new techniquethat recognizes musical instruments in polyphonic musical audio signals without using
onset detection or F0 estimation as explicit
and deterministic preprocesses. The key concept underlying our technique is to visualize
the probability that the sound of each target instrument exists at each time and witheach F0 as a spectrogram-like representation
called an instrogram. This probability is deﬁned as the product of two kinds of probabilities, called nonspecific instrument existence
probability and conditional instrument existence probability, which are calculated using thePreFEst 10) and hidden Markov models, respectively. The advantage of our technique is thaterrors due to the calculation of one probability do not inﬂuence the calculation of the otherprobability because the two probabilities can becalculated independently.
In addition, we describe the application of
the instrogram technique to MPEG-7 annotation and MIR based on instrumentation similarity. To achieve the annotation, we introduced two kinds of tags to the MPEG-7 standard. The ﬁrst is designed to describe the probabilities directly, and the second is designed
to obtain a symbolic representation such as an
event whereby a piano sound occurs at time t0
and continues until t1 . Such a representation
can be obtained using the Viterbi search on a
Markov chain with states that correspond to
instruments. To achieve the MIR based on instrumentation similarity, the distance (dissimilarity) between two instrograms is calculated

215

using dynamic time warping (DTW). A simple prototype system of similarity-based MIR
is also achieved based on our instrumentationbased similarity measure.
2. Instrogram
The instrogram is a spectrogram-like graphical representation of a musical audio signal,
which is useful for determining which instruments are used in the signal. In a basic format,
an instrogram correponds to a speciﬁc instrument. The instrogram has horizontal and vertical axes representing time and frequency, andthe intensity of the color of each point (t, f )shows the probability p(ωi ; t, f ) that the target
instrument ωi is used at time t and at an F0of f . An example is presented in Fig. 1. This
example shows the results of analyzing an audio signal of “Auld Lang Syne” played on the
piano, violin, and ﬂute. The target instruments
for analysis were the piano, violin, clarinet, andﬂute. If the instrogram is too detailed for some
purposes, it can be simpliﬁed by dividing theentire frequency region into a number of subregions and merging the results within each subregion. A simpliﬁed version of Fig. 1 is given
in Fig. 2. The original or simpliﬁed instrogramshows that the melodies in the high (approx.
note numbers 70–80), middle (60–75), and low(45–60) pitch regions are played on ﬂute, violin,
and piano, respectively.

Fig. 1 Example of the instrogram ☆ .

☆

Color versions of all ﬁgures are available at:
http://winnie.kuis.kyoto-u.ac.jp/˜kitahara/
instrogram/IPSJ07/

216

IPSJ Journal

Jan. 2007

Fig. 2 Simpliﬁed (summarized) instrogram for Fig. 1.

3. Algorithm for Calculating InstrogramLet Ω = {ω1 , · · · , ωm } be the set of target
instruments. We then have to calculate theprobability p(ωi ; t, f ) that a sound of the instrument ωi with an F0 of f exists at time tfor every target instrument ωi ∈ Ω. This probability is called the instrument existence probability (IEP). Here, we assume that multiple
instruments are not being played at the sametime and at the same F0, that is, ∀ωi , ωj ∈ Ω:i = j =⇒ p(ωi ∩ ωj ; t, f ) = 0. Let ω0 denote thesilence event, which means that no instruments
are being played, and let Ω+ = Ω ∪ {ω0 }. The
IEP then satisﬁes ωi ∈Ω+ p(ωi ; t, f ) = 1. Whenthe symbol “X” denotes the union event of all
target instruments, which stands for the existence of some instrument (i.e., X = ω1 ∪ · · · ∪ωm ), the IEP for each ωi ∈ Ω can be calculated
as the product of two probabilities:p(ωi ; t, f ) = p(X; t, f ) p(ωi |X; t, f ),because ωi ∩ X = ωi ∩ (ω1 ∪ · · · ∪ ωi ∪ · · · ∪ωm ) = ωi . Above, p(X; t, f ), called the nonspecific instrument existence probability (NIEP), is
the probability that the sound of some instrument with an F0 of f exists at time t, whilep(ωi |X; t, f ), called the conditional instrument
existence probability (CIEP), is the conditional
probability that, if the sound of some instrument with an F0 of f exists at time t, the inp(ω0 ; t, f ) isstrument is ωi . The probabilitygiven by p(ω0 ; t, f ) = 1 − ωi ∈Ω p(ωi ; t, f ).
3.1 Overview
Figure 3 shows an overview of the algorithm
for calculating an instrogram. Given an audio
signal, the spectrogram is ﬁrst calculated. The
short-time Fourier transform (STFT) shifted by

Fig. 3 Overview of our technique for calculating the
instrogram.

10 ms (441 points at 44.1 kHz sampling) with
an 8,192-point Hamming window is used in thecurrent implementation. We next calculate the
NIEPs and CIEPs. The NIEPs are calculatedusing the PreFEst 10) . The PreFEst models the
spectrum of a signal containing multiple musical instrument sounds as a weighted mixture ofharmonic-structure tone models at each frame.

Vol. 48

No. 1

Instrogram: Probabilistic Representation of Instrument Existence

For CIEPs, on the other hand, the temporal
trajectory of the harmonic structure with every
F0 is modeled with L-to-R HMMs because thetemporal characteristics are important in recognizing instruments. Once the time series offeature vectors are obtained for every F0, thelikelihoods of the paths in the chain of HMMs
are calculated.
The advantage of this technique lies in thefact that p(ωi ; t, f ) can be estimated robustly
because the two constituent probabilities arecalculated independently and are then integrated by multiplication. In most previous
studies, the onset time and F0 of each note
were ﬁrst estimated, and then the instrument
for the note was identiﬁed by analyzing spectral components extracted based on the resultsof the note estimation. The upper limit of
the instrument identiﬁcation performance wastherefore bound by the precedent note estimation, which is generally diﬃcult and not robust
for polyphonic music ☆ . Unlike such a notewise
symbolic approach, our non-symbolic and nonsequential approach is more robust for polyphonic music.
3.2 Nonspecific Instrument Existence
Probability
The NIEP p(X; t, f ) is estimated by using thePreFEst on the basis of the maximum likelihood estimation without assuming the numberof sound sources in a mixture. The PreFEst,which was originally developed for estimating
F0s of melody and bass lines, consists of three
processes: the PreFEst-front-end for frequency
analysis, the PreFEst-core for estimating therelative dominance of every possible F0, and
the PreFEst-back-end for evaluating the temporal continuity of the F0. Because the problem to be solved here is not the estimation of
the predominant F0s as melody and bass lines,
but rather the calculation of p(X; t, f ) of every
possible F0, we use only the PreFEst-core.
The PreFEst-core models an observed power
spectrum as a weighted mixture of tone modelsp(x|F ) for every possible F0 F . The tone modelp(x|F ), where x is the log frequency, represents☆

We tested robustness with respect to onset errors in
identifying an instrument for every note using ourprevious method 9) . Giving errors following a normal distribution with a standard deviation of e [s]to onset times, we obtained the following results:
e = 0 e = 0.05 e = 0.10 e = 0.15 e = 0.20
71.4%
69.2%
66.7%
62.5%
60.5%

217

a typical spectrum of harmonic structures, andthe mixture density p(x; θ (t) ) is deﬁned as Fh(t)p(x; θ ) =w(t) (F )p(x|F )dF,
Flθ (t) = {w(t) (F )|Fl ≤ F ≤ Fh},where Fl and Fh denote the lower and upper
limits, respectively, of the possible F0 range,
and w(t) (F ) is the weight of a tone modelFhp(x|F ) that satisﬁes Fl w(t) (F )dF = 1. If we
can estimate the model parameter θ (t) such thatthe observed spectrum is likely to have been
generated from p(x; θ (t) ), the spectrum can beconsidered to be decomposed into harmonicstructure tone models and w(t) (F ) can be interpreted as the relative predominance of thetone model with an F0 of F at time t. We
can therefore calculate the NIEP p(X; t, f ) as
the weight w(t) (f ), which can be estimated using the Expectation-Maximization (EM) algorithm 10) . In the current implementation, we
use the tone model given byp(x|F ) = αNc(h)G(x; F +1200 log2 h, W),

h=1



1
(x − m)2
G(x; m, σ) = √
exp −
,
2σ 2
2πσ 2
where α is a normalizing factor, N = 16, W =17 cent, and c(h) = G(h; 1, 5.5). This tonemodel was also used in the earliest version of
the PreFEst 11) .
3.3 Conditional Instrument Existence
Probability
The following steps are performed for every
frequency f .
3.3.1 Harmonic Structure Extraction
The temporal trajectory H(t, f ) of the harmonic structure (10 harmonics) of which F0 is
f is extracted.
3.3.2 Feature Extraction
It is important to design features thatwill eﬀectively recognize musical instruments.
Although mel-frequency cepstrum coeﬃcents(MFCCs) and Delta MFCCs are commonlyused in the ﬁeld of speech recognition studies,
we should design features that are optimized
for musical instrument sounds because musicalinstrument sounds have complicated temporalvariations (e.g., amplitude and frequency modulations). We therefore designed the 28 featureslisted in Table 1 based on our previous stud-218

IPSJ Journal
Table 1 Overview of 28 features.

Spectral features1

Spectral centroid

2

Relative power of fundamental component

3 – 10Relative cumulative power from fundamental to i-th components (i =2, 3, · · · , 9)11
Relative power in odd and even components
12 – 20
Number of components haveing a duration that is p% longer than the longest
duration (p = 10, 20, · · · , 90)
Temporal features
21

Gradient of straight line approximating
power envelope
22 – 24
Temporal mean of diﬀerentials of powerenvelope from t to t + iT /3 (i = 1, · · · , 3)
Modulation features
25 , 26

Amplitude and frequency of AM

27 , 28

Amplitude and frequency of FMies 9) . For every time t (every 10 ms in the implementation), we ﬁrst excerpt a T -length bitof the harmonic-structure trajectory Ht (τ, f )(t ≤ τ < t + T ) from the entire trajectory
H(t, f ) and then extract a feature vector x(t, f )consisting of the 28 features from Ht (τ, f ). The
dimensionality is then reduced to 12 dimensions
using principal component analysis with a proportion value of 95%. T is 500 ms in the current
implementation.
3.3.3 Probability Calculation
We train L-to-R HMMs, each consisting of 15
states ☆ , for target instruments ω1 , · · · , ωm , andthen basically consider the time series of feature vectors, {x(t, f )}, to be generated from a
Markov chain of these HMMs. Then, the CIEPp(ωi |X; t, f ) is calculated asp(x(t, f )|Mi )p(Mi ),p(Mi |x(t, f )) = mp(x(t, f )|Mi )p(Mi )i=1where Mi is the HMM corresponding to the instrument ωi . p(x(t, f )|Mi ) is trained from dataprepared in advance, and p(Mi ) is the a priori
probability.
In the above formulation, p(ωi |X; t, f ) forsome instruments may become greater thanzero even if no instruments are played. Theoretically, this does not matter because p(X; t, f )becomes zero in such cases. In practice, how☆

We used more states than those used in usual speech
recognition studies (typically three) because thenotes of musical instruments usually have longer durations than phonemes.

Jan. 2007

ever, p(X; t, f ) may not be zero, especially whena certain instrument is played at an F0 of an integer multiple or factor of f . To avoid this,
we prepare an HMM, M0 , trained with feature vectors extracted from silent signals (notethat some instruments may be played at nontarget F0s) and consider {x(t, f )} to be generated from a Markov chain of the m + 1 HMMs
(M0 , M1 , · · · , Mm ). The CIEP is therefore calculated asp(x(t, f )|Mi )p(Mi ),p(Mi |x(t, f )) = mp(x(t, f )|Mi )p(Mi )i=0where we use p(Mi ) = 1/(m + 1).
3.4 Simplifying Instrograms
Although we calculate IEPs for every F0,
some applications do not need such detailed
results. If the instrogram is used for retrieving musical pieces that include a certain instrument’s sounds, for example, IEPs for rough frequency regions (e.g., high, middle and low) are
suﬃcient. We therefore divide the entire frequency region into N subregions I1 , · · · , IN andcalculate the IEP p(ωi ; t, Ik ) for the k-th frequency subregion Ik . Here, this is deﬁned asp(ωi ; t, f ∈Ik f ), which can be obtained by iteratively calculating the following equation because the frequency axis is practically discrete.
p(ωi ; t, f1 ∪ · · · ∪ fi ∪ fi+1 )= p(ωi ; t, f1 ∪ · · · ∪ fi ) + p(ωi ; t, fi+1 )−p(ωi ; t, f1 ∪ · · · ∪ fi ) p(ωi ; t, fi+1 ),where Ik = {f1 , · · · , fi , fi+1 , · · · , fnk }.
4. Application
Here, we discuss the application of instrograms to MPEG-7 annotation and MIR.
4.1 MPEG-7 Annotation
There are two choices for transforming instrograms to MPEG-7 annotations. First, we cansimply represent IEPs as a time series of vectors. Because the MPEG-7 standard has no tag
for the instrogram annotation, we added several
original tags, as shown in Fig. 4. This example
shows the time series of eight-dimensional IEPs
for the piano (line 16) with a time resolutionof 10 ms (line 6). Each dimension corresponds
to a diﬀerent frequency subregion, which is deﬁned by dividing the entire range from 65.5 Hz
to 1,048 Hz (line 3) by 1/2 octave (line 4).
Second, we can transform instrograms into asymbolic representation. We also added several

Vol. 48

No. 1

Instrogram: Probabilistic Representation of Instrument Existence

1:<AudioDescriptor
2:
xsi:type="AudioInstrogramType"3:
loEdge="65.5" hiEdge="1048"
4:
octaveResolution="1/2">
5: <SeriesOfVector totalNumOfSamples="5982"
6:vectorSize="8" hopSize="PT10N1000F">
7:<Raw mpeg7:dim="5982 8">8:
0.0 0.0 0.0 0.0 0.718 0.017 0.051 0.0
9:0.0 0.0 0.0 0.0 0.724 0.000 0.085 0.0
10:0.0 0.0 0.0 0.0 0.702 0.013 0.089 0.0
11:0.0 0.0 0.0 0.0 0.661 0.017 0.063 0.0
12:......
13:</Raw>
14: </SeriesOfVector>
15: <SoundModel
16:
SoundModelRef="IDInstrument:Piano"/>17:</AudioDescriptor>

Fig. 4 Excerpt of example of instrogram annotation.

1:<MultimediaContent xsi:type="AudioType">
2: <Audio xsi:type="AudioSegmentType">
3:<MediaTime>
4:<MediaTimePoint>T00:00:06:850N1000
5:</MediaTimePoint>
6:<MediaDuration>PT0S200N1000
7:</MediaDuration>
8:</MediaTime>
9:<AudioDescriptor xsi:type="SoundSource"
10:
loEdge="92" hiEdge="130">
11:<SoundModel
12:SoundModelRef="IDInstrument:Piano"/>13:</AudioDescriptor>
14: </Audio>......

Fig. 5 Excerpt of example of symbolic annotation.

original tags, as shown in Fig. 5. This example
shows that an event for the piano (line 12) at a
pitch between 92 and 130 Hz (line 10) occurs at
6.850 s (line 4) and continues for 0.200 s (line 6).
To obtain this symbolic representation, we haveto estimate the event occurrence and its duration within every frequency subregion Ik . We
therefore obtain the time series of the instrument maximizing p(ωi ; t, Ik ) and then consider
it to be an output of a Markov chain, states ofwhich are ω0 , ω1 , · · · , ωm . (Fig. 6). The transition probabilities in the chain from a state
to the same state, from non-silence states tothe silence state, and from the silence state tonon-silence states are greater than zero, and theother probabilities are zero. After obtaining themost likely path in the chain, we can estimate
the occurrence and duration of an instrument
ωi from the transitions between the states ω0and ωi . This method assumes that only one instrument is played at the same time in each frequency subregion. When multiple instruments
are played in the same subregion at the sametime, the most predominant instrument will be219

Fig. 6 Markov chain model used in symbolic annotation. The values are transition probabilities,
where pst is the probability of staying at thesame state at the next time, which was experimentally determined as 1 − 10−16 .

annotated.
4.2 Music Information Retrieval basedon Instrumentation Similarity
One of the advantages of the instrogram,which is a non-symbolic representation, is toprovide a new instrumentation-based similarity measure. The similarity between two instrograms enables the MIR based on instrumentation similarity. As we pointed out in the
Introduction, this key technology is important
for automatic playlist generation and contentbased music recommendation. Here, insteadof calculating similarity, we calculate the distance (dissimilarity) between instrograms byusing DTW as follows:( 1 ) A vector pt for every time t is obtainedby concatenating the IEPs of all instruments:
pt = (p(ω1;t,I1 ), p(ω1;t,I2 ), · · · , p(ωm;t,IN )) ,where  is the transposition operator.
( 2 ) The distance between two vectors, p andq, is deﬁned as the cosine distance:dist(p, q) = 1 − (p, q)/||p||·||q||,
(p, p).
where (p, q) = p Rq, and ||p|| =R = (rij ) is a positive deﬁnite symmetric matrix. By setting rij with respect to related elements, e.g., violin vs. viola, to a value greaterthan zero, one can consider the similarity between such related instruments. When R is
the unit matrix, (p, q) and ||p|| are equivalent
to the standard inner product and norm.
( 3 ) The distance (dissimilarity) between{pt } and {q t } is calculated by applying DTW
with the above-mentioned distance measure.
The timbral similarity was also used in previous
MIR-related studies 12),13) . The timbral similarity was calculated on the basis of spectral features, such as MFCCs, directly extracted from

220

IPSJ Journal

Jan. 2007(a) FL–VN–PF(b) VN–CL–PF(c) PF–PF–PF(d) PF–CL–PF(e) PF–VN–PF(f) VN–VN–PF

Fig. 7 Results of calculating instrograms from “Auld Lang Syne” with six
diﬀerent instrumentations. “FL–VN–PF” means that the treble, middle, and bass parts are played on ﬂute, violin, and piano, respectively.

complex sound mixtures. Such features sometimes do not clearly reﬂect actual instrumentation, as will be implied in the next section,
because they are inﬂuenced not only by instrument timbres but also by arrangements, including the voicing of chords. On the other hand,because instrograms directly represent instrumentation, this will facilitate the appropriate
calculation of the similarity of instrumentation.
Moreover, instrograms have the following advantages:
Intuitiveness The musical meaning is intuitively clear.
Controllability By appropriately setting R,
users can ignore the diﬀerences between pitch
regions within the same instrument and/or
the diﬀerence between instruments within thesame instrument family.
5. Experiments
We conducted experiments on obtaining instrograms and symbolic annotations for bothaudio data generated on a computer and therecordings of real performances. In addition,
we tested the calculation of the similarity between instrograms for the real performances.
5.1 Use of Generated Audio Data
We ﬁrst conducted experiments on obtain-

ing instrograms from audio signals of trio music “Auld Lang Syne” used by Kashino et al 3) .
The audio signals were generated by mixing audio data from RWC-MDB-I-2001 14) (Variation
No. 1) according to a standard MIDI ﬁle (SMF)
that we input using a MIDI sequencer based on
Kashino’s score. The target instruments were
the piano (PF), violin (VN), clarinet (CL), andﬂute (FL). The training data for these instruments were taken from the audio data in RWCMDB-I-2001 with Variation Nos. 2 and 3. The
time resolution was 10 ms, and the frequency
resolution was every 100 cent. The width ofeach frequency subregion for the simpliﬁcation
was 600 cent. We used HTK 3.0 for HMMs.
The results are shown in Fig. 7. When wecompare (a) and (b), (a) has high IEPs for theﬂute in high-frequency regions while (b) hasvery low (almost zero) IEPs. In contrast, (a)has very low (almost zero) IEPs for the clarinetand (b) has high IEPs. Also, (d) has high IEPs
for the clarinet and almost zero IEPs for theviolin whereas (e) has high IEPs for the violin
and almost zero IEPs for the clarinet. In thecase of (c), the IEPs only for the piano are sufﬁciently high. Although both (e) and (f) areplayed on the piano and violin, the IEPs forthe violin in the highest frequency region areVol. 48

No. 1

Instrogram: Probabilistic Representation of Instrument Existence

diﬀerent. This correctly reﬂects the diﬀerence
between the actual instrumentations.
Based on the instrograms obtained above, we
conducted experiments on symbolic annotation
using the method described in Section 4.1. We
ﬁrst prepared ground truth (correct data) fromthe SMF used to generate the audio signals andthen evaluated the results based on the recall
rate R and precision rate P given by

N 
m 

R=

i=1 k=1i=1 k=1

Table 2 Results of symbolic annotation for “Auld
Lang Syne.”
FL–CL–PF
FL–PF–PF
FL–VN–PF
PF–CL–PF
PF–PF–PF
PF–VN–PF
VN–CL–PF
VN–PF–PF
VN–VN–PF
Average# frames correctly
annotated as ωi at Ik


m 
N 

# frames that should be

i=1 k=1m N 
i=1 k=1# frames annotatedas ωi at Ik

Precision
63.4%
89.4%
89.5%
79.3%
93.5%
76.6%
63.6%
76.1%
76.9%
78.7%

Table 3 Results of symbolic annotation for “Auld
Lang Syne”(all frequency subregions merged).


FL–CL–PF
FL–PF–PF
FL–VN–PF
PF–CL–PF
PF–PF–PF
PF–VN–PF
VN–CL–PF
VN–PF–PF
VN–VN–PF
Averageannotated as ωi at Ik

P =Recall
28.7%
38.5%
37.2%
22.2%
26.0%
24.2%
21.4%
14.3%
30.2%
27.0%

,

annotated as ωi at Ik

m 
N 

# frames correctly

221

 .

The results are shown in Table 2. We achieveda precision rate of 78.7% on average. Although
the recall rates were not high (14–38%), we consider the precision rates to be more important
than the recall rates for MIR; a system can use
recognition results even if some frames or frequency subregions are missing, whereas false results have a negative inﬂuence on MIR.
We also evaluated symbolic annotation by
merging all the frequency subregions; in otherwords, we ignored the diﬀerences between frequency subregions. This was because instrument annotation is useful even without F0 information for MIR. For example, a task such as
searching for piano solo pieces can be achieved
without F0 information. The evaluation was
conducted based on the recall rate R and precision rate P  . The recall and precision rates
for this evaluation are given by(#frames correctly annotated as ωi ),R =  i(#framesthat should be annotated as ωi )i(#frames correctly annotated as ωi )P= i .
i (#frames annotated as ωi )The results are listed in Table 3. The average
precision rate was 87.5% and the maximum was
95.4% for FL–VN–PF. The precision rates for all
pieces were over 80%, while the recall rates wereapproximately between 30 and 60%.
5.2 Use of Real Performances
We next conducted experiments on obtaining

Recall
36.0%
56.8%
44.5%
40.5%
62.2%
40.5%
29.2%
34.9%
40.8%
42.8%

Precision
80.3%
87.6%
95.4%
84.4%
91.4%
88.1%
87.6%
86.7%
85.3%
87.5%

Table 4 Musical pieces used and theirinstrumentations.
Classical
Jazz(i) No. 12, 14, 21, 38(ii) No. 19, 40(iii) No. 43(iv) No. 1, 2, 3

Strings
Piano+Strings
Piano+Flute
Piano solo

instrograms from the recordings of real performances of classical and jazz music taken fromthe RWC Music Database 15) . The instrumentation of all pieces is listed in Table 4. We onlyused the ﬁrst one-minute signal for each piece.
The experimental conditions were basically thesame as those in Section 5.1. Because the target
instruments were the piano, violin, clarinet, andﬂute, the IEPs for the violin should also be highwhen string instruments other than the violin
are played, and the IEPs for the clarinet should
always be low. The training data were takenfrom both RWC-MDB-I-2001 14) and NTTMSAP1 (a non-public musical sound database) ☆ .
The results, shown in Fig. 8, show that (a)and (b) have high IEPs for the violin while☆

The database called NTTMSA-P1 consists of isolated monophonic tones played by two diﬀerent individuals for each instrument. Every semitone overthe pitch range is played with three diﬀerent intensities for each instrument.

222

IPSJ Journal

Jan. 2007(a) RM-C No. 12 (Str.)(b) RM-C No. 14 (Str.)(c) RM-C No. 40 (Pf.+Str.)(d) RM-C No. 43 (Pf.+Fl.)(e) RM-J No. 1 (Pf.)(f) RM-J No. 2 (Pf.)

Fig. 8 Results of calculating instrograms from audio signals of real performances. RM-C stands for RWC-MDB-C-2001 and RM-J stands for
RWC-MDB-J-2001.

(e) and (f) have high IEPs for the piano. For(c), the IEPs for the violin increase after 10 sec,
whereas those for the piano are initially high.
This reﬂects the actual performances of these
instruments. When (d) is compared to (e) and(f), the former has slightly higher IEPs for theﬂute than the latter, although the diﬀerence isunclear. In general, the IEPs are not as clear as
those for signals generated by copy-and-pasting
waveforms of RWC-MDB-I-2001. This is because the acoustic characteristics of real performances have greater variety. This could be
improved by adding appropriate training data.
We also evaluated the symbolic annotation
for these real-performance recordings. The
evaluation was only conducted for the case inwhich all frequency subregions were merged because it is diﬃcult to manually prepare a reliable ground truth for each frequency subregion ☆ . The results are listed in Table 5. The
average precision rate was 69.4% and the maximum was 84.3%, which were lower than thosefor synthesized music. This would also be because of the great variety in the acoustic charac☆

Although the SMF corresponding to each piece isavailable in the RWC Music Database, it cannot be
used because the SMF and audio signal are not synchronized.

Table 5 Results of symbolic annotation for real
recordings (all frequency subregions merged).
RM-C No. 12
14
19
21
38
40
43
RM-J No. 1
2
3
Average

Recall
78.0%
76.0%
45.1%
89.9%
65.1%
50.8%
49.7%
62.1%
75.6%
45.9%
63.8%

Precision
63.4%
74.0%
65.6%
70.0%
64.0%
71.5%
84.3%
72.0%
69.3%
59.7%
69.4%

teristics of real performances. The recall rates,in contrast, were higher than those for synthesized music because the same instrument wasoften simultaneously played over multiple frequency subregions, in which the instrument was
regarded as correctly recognized if it was recognized in any of these subregions.
5.3 Similarity Calculation
We tested the calculation of the dissimilarities between instrograms. The results, listed inTable 6 (a), can be summarized as follows:• The dissimilarities within each group weregenerally less than 7,000 (except Group (ii)).
• Those between Groups (i) (played on
strings) and (iv) (piano) were generally

Vol. 48

No. 1

Instrogram: Probabilistic Representation of Instrument Existence223

Table 6 Dissimilarities in Instrumentation between musical pieces.
(a) Using IEPs (instrograms)
(i)
C12
C14
C21
C38
C19
C40
C43
J01
J02
J03

C12
0
6429
5756
7073
7320
8650
8910
9711
9856
9134

C14

C21

C38

C19

(ii)
C40

(iii)
C43

J01

(iv)
J02

J03

0
5734
6553
8181
8353
9635
10226
10125
9136

0
6411
7274
8430
9495
10252
10033
8894

0
7993
8290
9729
10324
10610
9376

0
8430
8148
8305
8228
8058

0
8235
8214
8139
8327

0
6934
7216
7480

0
6397
6911

0
7223

0

3-best-similarity
piecesC21, C14, C38
C21, C12, C38
C14, C12, C38
C21, C14, C38
C21, C12, C38
J02, J01, C43
J01, J02, J03
J02, J03, C43
J01, C43, J03
J01, J02, C43(b) Using MFCCs
(i)
C12
C14
C21
C38
C19
C40
C43
J01
J02
J03

C12
0
17733
17194
18500
17510
17417
18338
17657
17484
17799

C14

C21

C38

C19

(ii)
C40

(iii)
C43

J01

(iv)
J02

J03

0
18134
18426
18759
19011
17459
17791
17776
18063

0
18061
18222
18189
17728
17284
17359
17591

0
19009
19099
18098
17834
18009
18135

0
18100
18746
18133
17415
17814

0
18456
17983
17524
18038

0
16762
17585
17792

0
15870
16828

0
16987

0

3-best-similarity
piecesC21, C40, J02
C43, C12, J02
C12, J01, J02
J01, J02, C21
J02, C12, J03
C12, J02, J01
J01, C14, J02
J02, C43, J03
J01, J03, C21
J01, J02, C21

Note: “C” and “12” of “C12”, for example, represents a database (Classical/Jazz) and a piece number, respectively.

greater than 9,000, and some were greaterthan 10,000.
• Those between Groups (i) and (iii) (piano+ﬂute) were also around 9,000.
• Those between Groups (i) and (ii) (piano+strings), (ii) and (iii), and (ii) and (iv)were around 8,000. As one instrument iscommonly used in these pairs, these dissimilarities were reasonable.
• Those between Groups (iii) and (iv) werearound 7,000. Because the diﬀerence between these groups is only the presence ofthe ﬂute, these were also reasonable.
For comparison, Table 6 (b) lists the results
obtained using MFCCs. The 12-dimensional
MFCCs were extracted every 10 ms with a 25ms Hamming window. No Delta MFCCs wereused. After the MFCCs were extracted, thedissimilarity was calculated using the method
described in Section 4.2, where {pt } was asequence of 12-dimensional MFCC vectors instead of IEP vectors. Comparing the resultswith the two methods, we can see the following
diﬀerences:• The dissimilarities within Group (i) andthe dissimilarities between Group (i) andthe others for IEPs diﬀered more thanthose for MFCCs. In fact, all the three-

best-similarity pieces from those in Group(i) belonged to the same group, i.e., (i),for IEPs, while those for MFCCs contained
pieces out of Group (i).
• None of the three-best-similarity pieces
from the four pieces without strings(Groups (iii) and (iv)) contained strings for
IEPs, whereas those for MFCCs contained
pieces with strings (C14, C21).
We also developed a simple prototype systemthat searches pieces that have similar instrumentation to that speciﬁed by the user. The
demonstration of our MIR system and othermaterials will be available at: http://winnie.
kuis.kyoto-u.ac.jp/˜kitahara/instrogram/IPSJ07/.

6. Discussion
This study makes three major contributions
to instrument recognition and MIR.
The ﬁrst is the formulation of instrument
recognition as the calculation of NIEPs and
CIEPs. Because the calculation of NIEPs includes a process that can be considered to be
an alternative to the estimation of onset times
and F0s, this formulation has made it possible to omit their explicit estimation, which isdiﬃcult for polyphonic music. Based on similar motivations, Vincent, et al. 7) and Essid, et224

IPSJ Journal

al. 8) proposed new instrument recognition techniques. Vincent, et al.’s technique involves bothtranscription and instrument identiﬁcation in asingle optimization procedure. This technique
is based on a reasonable formulation and isprobably eﬀective but has only been tested on
solo and duo excerpts. Essid, et al.’s technique
identiﬁes the instrumentation, instead of the
instrument for each part, from a pre-designed
possible-instrumentation list. This technique isbased on the standpoint that music usually has
one of several typical instrumentations. They
reported successful experimental results, but
identifying instrumentations other than those
prepared is impossible. Our instrogram technique, in contrast, has made it possible torecognize instrumentation without making any
assumptions about instrumentation for audio
data, including synthesized music and real performances that have various instrumentations.
The second contribution is the establishmentof a graphical representation of instrumentation. Previous studies on music visualization
were generally based on MIDI-level symbolic
information 16),17) . Although spectrograms andspecmurts 18) are useful for visualization of audio signals, it is diﬃcult to recognize instrumentation from them. Our instrogram representation provides visual information on instrumentation, which should be useful for generatingmusic thumbnails and other visualizations suchas animations for entertainment.
The third contribution is the achievement of
an MIR based on instrumentation similarity.
Although both timbral similarity calculationand instrument recognition have been actively
investigated, no attempts of calculating the instrumentation similarity on the basis of instrument recognition techniques have been made,because previous instrument recognition haveaimed to determine the instruments that are
played in given signals. The instrogram, which
represents instrumentation as a set of continuous values, is an eﬀective approach to designinga continous similarity measure.
7. Conclusion
We described a new instrogram representation obtained by using a new musical instrument recognition technique that explicitly
uses neither onset detection nor F0 estimation.
Whereas most previous studies ﬁrst estimatedthe onset time and F0 of each note and thenidentiﬁed the instrument for each note, ourJan. 2007

technique calculates the instrument existence
probability for each target instrument at eachpoint on the time-frequency plane. This nonnote-based approach made it possible to avoid
adverse inﬂuences caused by errors of the onset
detection and F0 estimation.
In addition, we presented methods for applying the instrogram technique to MPEG-7 annotation and the MIR based on instrumentation
similarity. Although the note-based outputs
of previous instrument recognition were diﬃcult to apply to continuous similarity calculation, our instrogram representation provided ameasure for similarities in instrumentation andachieved MIR based on this measure.
The instrogram is related to Goto’s study onmusic understanding based on the idea thatpeople understand music without mentally representing it as a score 19) . Goto claims that
good music descriptors should be musically intuitive, fundamental to a professional methodof understanding music, and useful for various
applications. We believe that the instrogramsatisﬁes these requirements and therefore intend to apply it to the professional score-based
music understanding as well as various applications including MIR.
Acknowledgments This research was supported in part by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Grant-in-Aid for Scientiﬁc Research and Informatics Research Center for Development of Knowledge Society Infrastructure(COE program of MEXT, Japan). We used
the RWC Music Database and NTTMSA-P1.
We would like to thank everyone who has contributed to building these databases and wish
to thank NTT Communication Science Laboratories for giving us permission to use NTTMSAP1. The experiment presented in the appendix
was conducted by Mr. Katsutoshi Itoyama. We
appreciate his cooperation. We would also liketo thank Dr. Shinji Watanabe (NTT) for his
valuable comments.
References
1) Martin, K.D.: Sound-Source Recognition: A
Theory and Computational Model, PhD Thesis, MIT (1999).
2) Kashino, K., Nakadai, K., Kinoshita, T. andTanaka, H.: Application of the Bayesian Probability Network to Music Scene Analysis, Computational Auditory Scene Analysis, Rosenthal,
D.F. and Okuno, H.G. (Eds.), pp.115–137,

Vol. 48

No. 1

Instrogram: Probabilistic Representation of Instrument Existence

Lawrence Erlbaum Associates (1998).
3) Kashino, K. and Murase, H.: A Sound
Source Identification System for Ensemble Music based on Template Adaptation and Music Stream Extraction, Speech Comm., Vol.27,
pp.337–349 (1999).
4) Kinoshita, T., Sakai, S. and Tanaka, H.: Musical Sound Source Identification based on Frequency Component Adaptation, Proc. IJCAICASA Workshop, pp.18–24 (1999).
5) Eggink, J. and Brown, G.J.: Application ofMissing Feature Theory to the Recognitionof Musical Instruments in Polyphonic Audio,
Proc. ISMIR (2003).
6) Eggink, J. and Brown, G.J.: Extracting
Melody Lines from Complex Audio, Proc. ISMIR, pp.84–91 (2004).
7) Vincent, E. and Rodet, X.: Instrument Identification in Solo and Ensemble Music using
Independent Subspace Analysis, Proc. ISMIR,
pp.576–581 (2004).
8) Essid, S., Richard, G. and David, B.: Instrument Recognition in Polyphonic Music, Proc.
ICASSP, Vol.III, pp.245–248 (2005).
9) Kitahara, T., Goto, M., Komatani, K., Ogata,
T. and Okuno, H.G.: Instrument Identificationin Polyphonic Music: Feature Weighting with
Mixed Sounds, Pitch-dependent Timbre Modeling, and Use of Musical Context, Proc. ISMIR, pp.558–563 (2005).
10) Goto, M.: A Real-time Music-scene-description
System: Predominant-F0 Estimation for Detecting Melody and Bass Lines in Real-world
Audio Signals, Speech Comm., Vol.43, No.4,
pp.311–329 (2004).
11) Goto, M.: A Robust Predominant-F0 Estimation Method for Real-time Detection of
Melody and Bass Lines in CD Recordings,
Proc. ICASSP, Vol.II, pp.757–760 (2000).
12) Tzanetakis, G. and Cook, P.: Musical GenreClassification of Audio Signals, IEEE Trans.
Speech Audio Process., Vol.10, No.5, pp.293–
302 (2002).
13) Aucouturier, J.-J. and Pachet, F.: Music Similarity Measure: What’s the Use?, Proc. ISMIR,
pp.157–163 (2002).
14) Goto, M., Hashiguchi, H., Nishimura, T.
and Oka, R.: RWC Music Database: Music Genre Database and Musical Instrument
Sound Database, Proc. ISMIR, pp.229–230(2003).
15) Goto, M., Hashiguchi, H., Nishimura, T. andOka, R.: RWC Music Database: Popular, Classical, and Jazz Music Databases, Proc. ISMIR,
pp.287–288 (2002).
16) Hiraga, R.: A Look of Performance, IEEE Visualization (2002).

225

17) Hiraga, R., Miyazaki, R. and Fujishiro, I.: Performance Visualization—A New Challenge toMusic through Visualization, ACM Multimedia(2002).
18) Sagayama, S., Takahashi, K., Kameoka, H.
and Nishimoto, T.: Specmurt Anasylis: A
Piano-roll-visualization of Polyphonic Musicby Deconvolution of Log-frequency Spectrum,Proc. SAPA (2004).
19) Goto, M.: Music Scene Description Project:
Toward Audio-based Real-time Music Understanding, Proc. ISMIR, pp.231–232 (2003).
20) Kameoka, H., Nishimoto, T. and Sagayama,S.: Harmonic-temporal-structured Clustering
via Deterministic Annealing EM Algorithm
for Audio Feature Extraction, Proc. ISMIR,
pp.115–122 (2005).

Appendix
A.1 Experiment on Notewise Instrument Recognition
This Appendix presents the results of an experiment on notewise instrument recognition.
The problem that we deal with here is to detect a note sequence played on a speciﬁed instrument. We ﬁrst estimate the onset timeand F0 of every note using harmonic-temporalstructured clustering (HTC) 20) . For every
note, we then calculate the likelihood that thenote would be played on the speciﬁed instrument using our previous instrument identiﬁcation method 9) . Finally, we select the note thathas the maximal likelihood every time. Notethat we assume that multiple notes are notsimultaneously played on the speciﬁed instrument. We used “Auld Lang Syne” played on
the piano, violin, and ﬂute as a test sample.
This was synthesized by mixing audio data fromthe RWC-MDB-I-2001 14) similarly to the experiment reported in Section 5.1. Only if a detected note is actually played on the speciﬁed
instrument and at the correct F0 (note number) and the error of the estimated onset timeis less than e [s], the note is judged to be correct. Recognition was evaluated through thefollowing:# correctly detected notes,R=# notes that should be detected# correctly detected notes
P =.
# detected notes
The results are shown in Table 7. Recognition
especially for the violin and ﬂute was not sufﬁciently accurate. Although the results of thisexperiment cannot be directly compared with226

IPSJ Journal

Table 7 Results of experiment on notewise approach.
e = 0.2
e = 0.4
e = 0.6
PF VN FL PF VN FL PF VN FL
R 59% 36% 31% 59% 42% 34% 60% 47% 48%
P 86% 26% 17% 86% 24% 19% 87% 27% 27%

those obtained using our approach because they
were evaluated in diﬀerent ways, we can see thatthe notewise approach is not robust.
(Received May 8, 2006)(Accepted October 3, 2006)(Online version of this article can be found in
the IPSJ Digital Courier, Vol.3, pp.1–13.)

Jan. 2007

Kazunori Komatani received his B.E., M.S. and Ph.D.
degrees from Kyoto University,
Japan, in 1998, 2000, and 2002.
He is currently an Assistant Professor at the Graduate School ofInformatics of Kyoto University,
Japan. He received the 2002 FIT Young Researcher Award and the 2004 IPSJ Yamashita
SIG Research Award, both from the Information Processing Society of Japan. His researchinterests include spoken dialogue systems.

Tetsuro Kitahara received
his B.S. from the Tokyo University of Science in 2002 and his
M.S. from Kyoto University in
2004. He is currently a Ph.D.
student at Graduate School ofInformatics of Kyoto University.
He has been a Research Fellow of the Japan Society for the Promotion of Science since 2005.
His research interests include music informatics. He has received several awards including
the TELECOM System Technology Award forStudent and the IPSJ 67th National Convention Best Paper Award for Young Researcher.

Tetsuya Ogata received his
B.E., M.S. and Ph.D. degrees
from Waseda University, Japan,
in 1993, 1995, and 2000. He
is currently an Associate Professor at the Graduate School ofInformatics of Kyoto University,
Japan. He served concurrently as a Visiting
Associate Professor of the Humanoid Robotics
Institute of Waseda University, and Vising Scientist of the Brain Science Institute of RIKEN.
His research interests include multi-modal active sensing and robot imitation.

Masataka Goto received
his Doctor of Engineering degree in electronics, information,and communication engineeringfrom Waseda University, Japan,
in 1998. He is currently a Senior Research Scientist of the
National Institute of Advanced Industrial Science and Technology (AIST). He served concurrently as a Researcher in Precursory Research for Embryonic Science and Technology (PRESTO), Japan Science and Technology
Corporation (JST) from 2000 to 2003, and has
been an Associate Professor of the Department
of Intelligent Interaction Technologies, Graduate School of Systems and Information Engineering, University of Tsukuba, since 2005.
He has received 18 awards, including the IPSJBest Paper Award, IPSJ Yamashita SIG Research Awards, and Interaction 2003 Best Paper Award.

Hiroshi G. Okuno received
his B.A. and Ph.D. from the
University of Tokyo in 1972 and1996. He worked for NTT, JSTand the Tokyo University of Science. He is currently a professor of in the Graduate School ofInformatics at Kyoto University. He is currently engaged in computational auditory scene
analysis, music scene analysis and robot audition. He has received various awards including the 1990 Best Paper Award of JSAI,
the Best Paper Award of IEA/AIE-2001 and2005, and IEEE/RSJ Nakamura Award for
IROS-2001 Best Paper Nomination Finalist.
He was also awarded 2003 Funai Information Science Achievement Award. He editedwith David Rosenthal “Computational Auditory Scene Analysis” (LEA, 1998) and with Taiichi Yuasa “Advanced Lisp Technology” (T&F,
2002).