Hindawi Publishing Corporation
EURASIP Journal on Advances in Signal Processing
Volume 2007, Article ID 51979, 15 pages
doi:10.1155/2007/51979

Research Article
Instrument Identification in Polyphonic Music: Feature
Weighting to Minimize Influence of Sound Overlaps
Tetsuro Kitahara,1 Masataka Goto,2 Kazunori Komatani,1 Tetsuya Ogata,1 and Hiroshi G. Okuno1
1 Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University, Sakyo-Ku,

Kyoto 606-8501, Japan
2 National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki 305-8568, Japan

Received 7 December 2005; Revised 27 July 2006; Accepted 13 August 2006
Recommended by Ichiro Fujinaga
We provide a new solution to the problem of feature variations caused by the overlapping of sounds in instrument identificationin polyphonic music. When multiple instruments simultaneously play, partials (harmonic components) of their sounds overlap
and interfere, which makes the acoustic features diﬀerent from those of monophonic sounds. To cope with this, we weight features
based on how much they are aﬀected by overlapping. First, we quantitatively evaluate the influence of overlapping on each featureas the ratio of the within-class variance to the between-class variance in the distribution of training data obtained from polyphonicsounds. Then, we generate feature axes using a weighted mixture that minimizes the influence via linear discriminant analysis. In
addition, we improve instrument identification using musical context. Experimental results showed that the recognition rates using both feature weighting and musical context were 84.1% for duo, 77.6% for trio, and 72.3% for quartet; those without using
either were 53.4, 49.6, and 46.5%, respectively.
Copyright © 2007 Tetsuro Kitahara et al. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly
cited.

1.

INTRODUCTION

While the recent worldwide popularization of online music
distribution services and portable digital music players has
enabled us to access a tremendous number of musical excerpts, we do not yet have easy and eﬃcient ways to findthose that we want. To solve this problem, eﬃcient music information retrieval (MIR) technologies are indispensable. In
particular, automatic description of musical content in a universal framework is expected to become one of the most important technologies for sophisticated MIR. In fact, frameworks such as MusicXML [1], WEDELMUSIC Format [2],and MPEG-7 [3] have been proposed for describing musicor multimedia content.
One reasonable approach for this music description isto transcribe audio signals to traditional music scores because the music score is the most common symbolic music representation. Many researchers, therefore, have tried
automatic music transcription [4–9], and their techniques
can be applied to music description in a score-based format
such as MusicXML. However, only a few of them have dealt
with identifying musical instruments. Which instruments areused is important information for two reasons. One is that itis necessary for generating a complete score. Notes for different instruments, in general, should be described on different staves in a score, and each stave should have a description of instruments. The other reason is that the instruments characterize musical pieces, especially in classical music. The names of some musical forms are based on instrument names, such as “piano sonata” and “string quartet.”
When a user, therefore, wants to search for certain types ofmusical pieces, such as piano sonatas or string quartets, a retrieval system can use information on musical instruments.
This information can also be used for jumping to the point
when a certain instrument begins to play.
This paper, for these reasons, addresses the problem ofwhich facilitates the above-mentioned score-based music annotation, in audio signals of polyphonic music, in particular, classical Western tonal music. Instrument identification
is a sort of pattern recognition that corresponds to speaker
identification in the field of speech information processing.
Instrument identification, however, is a more diﬃcult problem than noiseless single-speaker identification because, inmost musical pieces, multiple instruments simultaneously

2play. In fact, studies dealing with polyphonic music [7, 10–13] have used duo or trio music chosen from 3–5 instrument
candidates, whereas those dealing with monophonic sounds[14–23] have used 10–30 instruments and achieved the performance of about 70–80%. Kashino and Murase [10] reported a performance of 88% for trio music played on piano, violin, and flute given the correct fundamental frequencies (F0s). Kinoshita et al. [11] reported recognition rates ofaround 70% (70–80% if the correct F0s were given). Eggink
and Brown [13] reported a recognition rate of about 50% forduo music chosen from five instruments given the correct
F0s. Although a new method that can deal with more complex musical signals has been proposed [24], it cannot be applied to score-based annotation such as MusicXML because
the key idea behind this method is to identify instrumentation instead of instruments at each frame, not for each note.
The main diﬃculty in identifying instruments in polyphonicmusic is the fact that acoustical features of each instrument
cannot be extracted without blurring because of the overlapping of partials (harmonic components). If a clean sound foreach instrument could be obtained using sound separation
technology, the identification of polyphonic music would become equivalent to identifying the monophonic sound ofeach instrument. In practice, however, a mixture of sounds
is diﬃcult to separate without distortion.
In this paper, we approach the above-mentioned overlapping problem by weighting each feature based on how
much the feature is aﬀected by the overlapping. If we cangive higher weights to features suﬀering less from this problem and lower weights to features suﬀering more, it will facilitate robust instrument identification in polyphonic music. To do this, we quantitatively evaluate the influence ofthe overlapping on each feature as the ratio of the withinclass variance to the between-class variance in the distributionof training data obtained from polyphonic sounds because
greatly suﬀering from the overlapping means having large
variation when polyphonic sounds are analyzed. This evaluation makes the feature weighting described above equivalent to dimensionality reduction using linear discriminant
analysis (LDA) on training data obtained from polyphonicsounds. Because LDA generates feature axes using a weighted
mixture where the weights minimize the ratio of the withinclass variance to the between-class variance, using LDA on
training data obtained from polyphonic sounds generates a
subspace where the influence of the overlapping problem isminimized. We call this method DAMS (discriminant analysis with mixed sounds). In previous studies, techniques suchas time-domain waveform template matching [10], feature
adaptation with manual feature classification [11], and themissing feature theory [12] have been tried to cope with theoverlapping problem, but no attempts have been made togive features appropriate weights based on their robustness
to the overlapping.
In addition, we propose a method for improving instrument identification using musical context. This method isaimed at avoiding musically unnatural errors by considering the temporal continuity of melodies; for example, if theidentified instrument names of a note sequence are all “flute”

EURASIP Journal on Advances in Signal Processing
except for one “clarinet,” this exception can be considered anerror and corrected.
The rest of this paper is organized as follow. In Section 2,
we discuss how to achieve robust instrument identificationin polyphonic music and propose our feature weighting
method, DAMS. In Section 3, we propose a method for usingmusical context. Section 4 explains the details of our instrument identification method, and Section 5 reports the resultsof our experiments including those under various conditions
that were not reported in [25]. Finally, Section 6 concludes
the paper.
2.

INSTRUMENT IDENTIFICATION ROBUSTTO OVERLAPPING OF SOUNDSIn this section, we discuss how to design an instrument identification method that is robust to the overlapping of sounds.
First, we mention the general formulation of instrument
identification. Then, we explain that extracting harmonic
structures eﬀectively suppresses the influence of other simultaneously played notes. Next, we point out that harmonic
structure extraction is insuﬃcient and we propose a methodof feature weighting to improve the robustness.
2.1.

General formulation of instrument identification

In our instrument identification methodology, the instrument for each note is identified. Suppose that a given audio
signal contains K notes, n1 , n2 , . . . , nk , . . . , nK . The identification process has two basic subprocesses: feature extraction
and a posteriori probability calculation. In the former process, a feature vector consisting of some acoustic features is
extracted from the given audio signal for each note. Let xk be
the feature vector extracted for note nk . In the latter process,for each of the target instruments, ω1 , . . . , ωm , the probability p(ωi | xk ) that the feature vector xk is extracted from asound of the instrument ωi is calculated. Based on the Bayes
theorem, p(ωi | xk ) can be expanded as follows:


  



p xk | ωi p ωi  ,
p ωi | xk = m 
j =1 p xk | ω j p ω j(1)where p(xk | ωi ) is a probability density function (PDF) andp(ωi ) is the a priori probability with respect to the instrument ωi . The PDF p(xk | ωi ) is trained using data preparedin advance. Finally, the name of the instrument maximizingp(ωi | xk ) is determined for each note nk . The symbols usedin this paper are listed in Table 1.
2.2.

Use of harmonic structure modelIn speech recognition and speaker recognition studies, features of spectral envelopes such as Mel-frequency cepstrum
coeﬃcients are commonly used. Although they can reasonably represent the general shapes of observed spectra, when asignal of multiple instruments simultaneously playing is analyzed, focusing on the component corresponding to each
instrument from the observed spectral envelope is diﬃcult.
Because most musical sounds except percussive ones have

Tetsuro Kitahara et al.

3

Table 1: List of symbols.
n1 , . . . , nK

Notes contained in a given signal

xk

Feature vector for note nk

ω1 , . . . , ωm

Target instrumentsp(ωi | xk )A posteriori probabilityp(ωi )A priori probabilityp(xk | ωi )Probability density function
Maximum number of simultaneously playedsh (nk ), sl (nk )notes in higher or lower pitch ranges when note
nk is being playedN

Set of notes extracted for contextcNumber of notes in N

fFundamental frequency (F0) of a given notefx

F0 of feature vector xµi ( f )F0-dependent mean function for instrument ωi

Σi

F0-normalized covariance for instrument ωi

χi

Set of training data of instrument ωip(x | ωi ; f )Probability density function for F0-dependent

D2 (x; µi ( f ), Σi )Squared Mahalanobis distancemultivariate normal distribution

harmonic structures, previous studies on instrument identification [7, 9, 11] have commonly extracted the harmonic
structure of each note and then extracted acoustic featuresfrom the structures.
We also extract the harmonic structure of each note andthen extract acoustic features from the structure. The harmonic structure model H (nk ) of the note nk can be represented as the following equation:

H nk =Fi (t), Ai (t) | i = 1, 2, . . . , h, 0 ≤ t ≤ T , (2)where Fi (t) and Ai (t) are the frequency and amplitude of theith partial at time t. Frequency is represented by relative frequency where the temporal median of the fundamental frequency, F1 (t), is 1. Above, h is the number of harmonics, andT is the note duration. This modeling of musical instrument
sounds based on harmonic structures can restrict the influence of the overlapping of sounds of multiple instruments tothe overlapping of partials. Although actual musical instrument sounds contain nonharmonic components, which canbe factors characterizing sounds, we focus only on harmonicones because nonharmonic ones are diﬃcult to reliably extract from a mixture of sounds.
2.3. Feature weighting based on robustness
to overlapping of sounds
As described in the previous section, the influence of theoverlapping of sounds of multiple instruments is restricted
to the overlapping of the partials by extracting the harmonic

structures. If two notes have no partials with common frequencies, the influence of one on the other when the twonotes are simultaneously played may be ignorably small. In
practice, however, partials often overlap. When two notes
with the pitches of C4 (about 262 Hz) and G4 (about 394 Hz)
are simultaneously played, for example, the 3 ith partials of
the C4 note and the 2 ith partials of the G4 note overlap forevery natural number i. Because note combinations that cangenerate harmonious sounds cause overlaps in many partials
in general, coping with the overlapping of partials is a serious
problem.
One eﬀective approach for coping with this overlapping
problem is feature weighting based on the robustness to theoverlapping problem. If we can give higher weights to features suﬀering less from this problem and lower weights tofeatures suﬀering more, it will facilitate robust instrument
identification in polyphonic music. Concepts similar to thisfeature weighting, in fact, have been proposed, such as themissing feature theory [12] and feature adaptation [11].
(i) Eggink and Brown [12] applied the missing feature
theory to the problem of identifying instruments in polyphonic music. This is a technique for canceling unreliablefeatures at the identification step using a vector called a mask,
which represents whether each feature is reliable or not. Because masking a feature is equivalent to giving a weight ofzero to it, this technique can be considered an implementation of the feature weighting concept. Although this technique is known to be eﬀective if the features to be masked are
given, automatic mask estimation is very diﬃcult in generaland has not yet been established.
(ii) Kinoshita et al. [11] proposed a feature adaptation
method. They manually classified their features for identification into three types (additive, preferential, and fragile) according to how the features varied when partials overlapped.
Their method recalculates or cancels the features extracted
from overlapping components according to the three types.
Similarly to Eggink’s work, canceling features can be considered an implementation of the feature weighting concept. Because this method requires manually classifying features inadvance, however, using a variety of features is diﬃcult. They
introduced a feature weighting technique, but this technique
was performed on monophonic sounds, and hence did notcope with the overlapping problem.
(iii) Otherwise, there has been Kashino’s work based ona time-domain waveform template-matching technique withadaptive template filtering [10]. The aim was the robust
matching of an observed waveform and a mixture of waveform templates by adaptively filtering the templates. This
study, therefore, did not deal with feature weighting basedon the influence of the overlapping problem.
The issue in the feature weighting described above is
how to quantitatively design the influence of the overlapping problem. Because training data were obtained only frommonophonic sounds in previous studies, this influence could
not be evaluated by analyzing the training data. Our DAMS
method quantitatively models the influence of the overlapping problem on each feature as the ratio of the within-classvariance to the between-class variance in the distribution

4

EURASIP Journal on Advances in Signal Processing

Frequency(Vn G4)Frequency

A mixtureof sounds[0.124, 0.634, . . .]Time

Harmonic
structure
extraction(Pf C4)

Featureextraction
Feature vector (Pf C4)

Frequency

Time

Feature vector (Vn G4)[0.317, 0.487, . . .]Time

Vn : violin
Pf : piano

Figure 1: Overview of process of constructing mixed-sound template.

of training data obtained from polyphonic sounds. As described in the introduction, this modeling makes weighting
features to minimize the influence of the overlapping problem equivalent to applying LDA to training data obtained
from polyphonic sounds.
Training data are obtained from polyphonic sounds
through the process shown in Figure 1. The sound of eachnote in the training data is labeled in advance with the instrument name, the F0, the onset time, and the duration. By
using these labels, we extract the harmonic structure corresponding to each note from the spectrogram. We then extract
acoustic features from the harmonic structure. We thus obtain a set of many feature vectors, called a mixed-sound template, from polyphonic sound mixtures.
The main issue in constructing a mixed-sound templateis to design an appropriate subset of polyphonic sound mixtures. This is a serious issue because there are an infinite
number of possible combinations of musical sounds due tothe large pitch range of each instrument.1 The musical feature that is the key to resolving this issue is a tendency of intervals of simultaneous notes. In Western tonal music, someintervals such as minor 2nds are more rarely used than otherintervals such as major 3rds and perfect 5ths because minor 2nds generate dissonant sounds in general. By generating
polyphonic sounds for template construction from the scoresof actual (existing) musical pieces, we can obtain a data set
that reflects the tendency mentioned above.2 We believe that
this approach improves instrument identification even if thepieces used for template construction are diﬀerent from thepiece to be identified for the following two reasons.
(i) There are diﬀerent distributions of intervals found insimultaneously sounding notes in tonal music. For example,
1 Because our data set of musical instrument sounds consists of 2651 notesof five instruments, C(2651, 3) ≈ 3.1 billion diﬀerent combinations are
possible even if the number of simultaneous voices is restricted to three.
About 98 years would be needed to train all the combinations, assumingthat one second is needed for each combination.
2 Although this discussion is based on tonal music, this may be applicable
to atonal music by preparing the scores of pieces of atonal music.

Figure 2: Example of musically unnatural errors. This example isexcerpted from results of identifying each note individually in apiece of trio music. Marked notes are musically unnatural errors,
which can be avoided by using musical context. PF, VN, CL, and FL
represent piano, violin, clarinet, and flute.

three simultaneous notes with the pitches of C4, C#4, and D4
are rarely used except for special eﬀects.
(ii) Because we extract the harmonic structure from eachnote, as previously mentioned, the influence of multiple instruments simultaneously playing is restricted to the overlapping of partials. The overlapping of partials can be explained by two main factors: which partials are aﬀected byother sounds, related to note combinations, and how mucheach partial is aﬀected, mainly related to instrument combinations. Note combinations can be reduced because ourmethod considers only relative-pitch relationships, and thelack of instrument combinations is not critical to recognition
as we find in an experiment described below. If the intervalsof note combinations in a training data set reflect those inactual music, therefore, the training data set will be eﬀective
despite a lack of other combinations.
3.

USE OF MUSICAL CONTEXTIn this section, we propose a method for improving instrument identification by considering musical context. The aimof this method is to avoid unusual events in tonal music, forexample, only one clarinet note appearing in a sequence ofnotes (a melody) played on a flute, as shown in Figure 2. As
mentioned in Section 2.1, the a posteriori probability p(ωi |xk ) is given by p(ωi | xk ) = p(xk | ωi )p(ωi )/ j p(xk |ω j )p(ω j ). The key idea behind using musical context is to
apply the a posteriori probabilities of nk ’s temporally neighboring notes to the a priori probability p(ωi ) of the note nk(Figure 3). This is based on the idea that if almost all notes
around the note nk are identified as the instrument ωi , nk isalso probably played on ωi . To achieve this, we have to resolve
the following issue.

Tetsuro Kitahara et al.

5
Assuming that the following notes are playedon the same instrument. . .
nk 2
nk 1
nk
nk+1

Issue: distinguishing notes played on the same instrument as
nk from neighboring notes
Because various instruments are played at the same time, an
identification system has to distinguish notes that are playedon the same instrument as the note nk from notes played onother instruments. This is not easy because it is mutually dependent on musical instrument identification.
We resolve this issue as follows.

A posteriori probabilities
p(ωi xk 2 ) p(ωi xk 1 ) p(ωi xk ) p(ωi xk+1 ) p(ωi xk+2 )Defined asp(xk ωi ) p(ωi )p(xk )

Solution: take advantage of the parallel movement ofsimultaneous parts.
In Western tonal music, voices rarely cross. This may be
explained due to the human’s ability to recognize multiple
voices easier if they do not cross each other in pitch [26].
When they listen, for example, to two simultaneous note sequences that cross, one of which is descending and the otherof which is ascending, they cognize them as if the sequences
approach each other but never cross. Huron also explainsthat the pitch-crossing rule (parts should not cross with respect to pitch) is a traditional voice-leading rule and can bederived from perceptual principles [27]. We therefore judge
whether two notes, nk and n j , are in the same part (i.e.,
played on the same instrument) as follows: let sh (nk ) andsl (nk ) be the maximum number of simultaneously playednotes in the higher and lower pitch ranges when the note nk
is being played. Then, the two notes nk and n j are considered to be in the same part if and only if sh (nk ) = sh (n j ) andsl (nk ) = sl (n j ) (Figure 4). Kashino and Murase [10] have introduced musical role consistency to generate music streams.
They have designed two kinds of musical roles: the highest and lowest notes (usually corresponding to the principal
melody and bass lines). Our method can be considered an
extension of their musical role consistency.
3.1. 1st pass: precalculation of a posteriori
probabilities
For each note nk , the a posteriori probability p(ωi | xk ) is
calculated by considering the a priori probability p(ωi ) to bea constant because the a priori probability, which depends
on the a posteriori probabilities of temporally neighboringnotes, cannot be determined in this step.
3.2. 2nd pass: recalculation of a posteriori
probabilities
This pass consists of three steps.

nk+2

A priori probability
Calculated based on a posteriori probabilities
of previous and following notes

Figure 3: Key idea for using musical context. To calculate a posteriori probability of note nk , a posteriori probabilities of temporally
neighboring notes of nk are used.

(2) Calculating a priori probability
The a priori probability of the note nk is calculated based on
the a posteriori probabilities of the notes extracted in the previous step. Let p1 (ωi ) and p2 (ωi ) be the a priori probabilities
calculated from musical context and other cues, respectively.
Then, we define the a priori probability p(ωi ) to be calculated
here as follows:
 

 

 

p ωi = λp1 ωi + (1 − λ)p2 ωi ,(3)

where λ is a confidence measure of musical context. Although
this measure can be calculated through statistical analysis as
the probability that the note nk will be played on instrument
ωi when all the extracted neighboring notes of nk are playedon ωi , we use λ = 1 − (1/2)c for simplicity, where c is thenumber of notes in N . This is based on the heuristics thatas more notes are used to represent a context, the context
information is more reliable. We define p1 (ωi ) as follows: 

p1 ωi =

1  
p ωi | x j ,α n j ∈N

(4)

where x j is the feature vector forthenote n j and α is thenormalizing factor given by α = ωi n j p(ωi | x j ). We use
p2 (ωi ) = 1/m for simplicity.
(3) Updating a posteriori probability
The a posteriori probability is recalculated using the a priori
probability calculated in the previous step.

(1) Finding notes played on the same instrument
Notes that satisfy {n j | sh (nk ) = sh (n j ) ∩ sl (nk ) = sl (n j )}are extracted from notes temporally neighboring nk . This extraction is performed from the nearest note to farther notes
and stops when c notes have been extracted (c is a positive
integral constant). Let N be the set of the extracted notes.

4.

DETAILS OF OUR INSTRUMENT
IDENTIFICATION METHOD

The details of our instrument identification method are
given below. An overview is shown in Figure 5. First, thespectrogram of a given audio signal is generated. Next, the6

EURASIP Journal on Advances in Signal Processing(0, 2)(0, 1)(0, 1)(0, 2)(0, 2)(0, 2)(0, 1) (1, 1)(0, 2)(0, 2)(0, 2)(1, 1) (1, 1)(1, 1)(1, 1)(1, 1)(1, 1)(1, 0) (1, 0) (1, 0)(1, 0)(1, 0)(2, 0)(2, 0) (2, 0)(2, 0) (2, 0) (2, 0) (2, 0) (2, 0)(2, 0) (2, 0) (2, 0)

A pair of notes that is correctly judged to be played on the same instrument
A pair of notes that is not judged to be played on the same instrument although it actually is

Figure 4: Example of judgment of whether notes are played on the same instrument. Each tuple (a,b) represents sh (nk ) = a and sl (nk ) = b.

harmonic structure of each note is extracted based on dataon the F0, the onset time, and the duration of each note,
which are estimated in advance using an existing method(e.g., [7, 9, 28]). Then, feature extraction, dimensionality reduction, a posteriori probability calculation, and instrument
determination are performed in that order.

450-millisecond template is selected for a 500-millisecond
note. In this paper, the 300-milliseconds, 450-millisecond,and 600-millisecond templates are called Template Types I,
II, and III. Notes shorter than 300 milliseconds are not identified.
4.3.

4.1. Short-time Fourier transform
The spectrogram of the given audio signal is calculated using
the short-time Fourier transform (STFT) shifted by 10 milliseconds (441 points at 44.1 kHz sampling) with an 8192point Hamming window.
4.2. Harmonic structure extraction
The harmonic structure of each note is extracted according tonote data estimated in advance. Spectral peaks corresponding to the first 10 harmonics are extracted from the onset
time to the oﬀset time. The oﬀset time is calculated by adding
the duration to the onset time. Then, the frequency of thespectral peaks is normalized so that the temporal mean of F0
is 1.
Next, the harmonic structure is trimmed because training and identification require notes with fixed durations. Because a mixed-sound template with a long duration is morestable and robust than a template with a short one, trimminga note to keep it as long as possible is best. We therefore prepare three templates with diﬀerent durations (300, 450, and
600 milliseconds), and the longest usable, as determined by
the actual duration of each note, is automatically selected
and used for training and identification.3 For example, theFeature extraction

Features that are useful for identification are extracted fromthe harmonic structure of each note. From a feature set that
we previously proposed [19], we selected 43 features (forTemplate Type III), summarized in Table 2, that we expected
to be robust with respect to sound mixtures. We use 37 features for Template Type II and 31 for I because of the limitations of the note durations.
4.4.

Dimensionality reduction

Using the DAMS method, the subspace minimizing the influence of the overlapping problem is obtained. Because a feature space should not be correlated to robustly perform the
LDA calculation, before using the DAMS method, we obtaina noncorrelative space by using principal component analysis (PCA). The dimensions of the feature space obtained with
PCA are determined so that the cumulative proportion valueis 99% (20 dimensions in most cases). By using the DAMS
method in this subspace, we obtain an (m − 1)-dimensional
space (m: the number of instruments in the training data).
4.5.

A posteriori probability calculation

For each note nk , the a posteriori probability p(ωi | xk ) is
calculated. As described in Section 2.1, this probability canbe calculated using the following equation:

3 The template is selected based on the fixed durations instead of the tempo

because temporal variations of spectra, which influence the dependencyof features on the duration, occur on the absolute time scale rather thanin the tempo.

p ωi | xk = 



  

p xk | ωi p ωi  .
j p xk | ω j p ω j(5)

Tetsuro Kitahara et al.

7

Table 2: Overview of 43 features.

Audio signal

Spectral features

STFT

1

Spectral centroid

2

Relative power of fundamental component

Spectrogram

Relative cumulative power from fundamental

3 – 10

to ith components (i = 2, 3, . . . , 9)11Musical notes
1) 0–1000 ms C42) 0–500 ms C23) 500–1000 ms G24) 1500–2000 ms C4

Harmonic structure extraction
1)

2)

3)

4)

Relative power in odd and even components

12 – 20

Number of components whose durations

Feat. extract.

Feat. extract.

Feat. extract.

Feat. extract.

are p% longer than the longest duration

Dim. reduct.

Dim. reduct.

Dim. reduct.

Dim. reduct.

A post. prob.
precalculation

A post. prob.
precalculation

A post. prob.
precalculation

A post. prob.
precalculation(p = 10, 20, . . . , 90)
Temporal featuresp(ωi x1 ) U: 1
L: 0

Gradient of straight line approximating

21

p(ωi x2 ) U: 0
L: 1p(ωi x3 ) U: 1
L: 0p(ωi x4 ) U: 0
L: 1

power envelope
Average diﬀerential of power envelope
22 – 30∗

during t-second interval from onset time

A post. prob.
recalculation

A post. prob.
recalculation

A post. prob.
recalculation

A post. prob.
recalculationp(ωi x1 )p(ωi x2 )p(ωi x3 )p(ωi x4 )Instr. determ.

Instr. determ.

Instr. determ.

Instr. determ.

ViolinPiano

ViolinPiano(t = 0.15, 0.20, 0.25, . . . , 0.55 (s))
31 – 39 ∗Ratio of power at t second after onset time

Modulation features
40 , 41

Amplitude and frequency of AM

42 , 43

Amplitude and frequency of FM

Feat. extract. : feature extraction
Dim. reduct. : dimensionality reduction
A post. prob. : a posteriori probability
Instr. determ. : instrument determination∗

In Template Types I and II, some of these features have been excluded dueto the limitations of the note durations.

The PDF p(xk | ωi ) is calculated from training data preparedin advance by using an F0-dependent multivariate normal
distribution, as it is defined in our previous paper [19]. The
F0-dependent multivariate normal distribution is designed
to cope with the pitch dependency of features. It is specified
by the following two parameters.
(i) F0-dependent mean function μi ( f )For each element of the feature vector, the pitch dependencyof the distribution is approximated as a function (cubic polynomial) of F0 using the least-square method.

The F0-normalized covariance is calculated using the following equation:
1
χi


x∈χi

 

x − µi f x

where χi is the set of the training data of instrument ωi , |χi |is the size of χi , fx denotes the F0 of feature vector x, and 
represents the transposition operator.
Once these parameters are estimated, the PDF is given as

p xk | ωi ; f =1(2π)d/2 Σi−

1/2 exp


1 2
D xk ; µi ( f ), Σi ,2(7)where d is the number of dimensions of the feature space andD2 is the squared Mahalanobis distance defined by












D2 xk ; µi ( f ), Σi = xk − µi ( f ) Σi−1 xk − µi ( f ) .

(8)

The a priori probability p(ωi ) is calculated on the basis of
the musical context, that is, the a posteriori probabilities ofneighboring notes, as described in Section 3.

(ii) F0-normalized covariance ΣiΣi =Figure 5: Flow of our instrument identification method.

 

x − µi f x

,

(6)

4.6.

Instrument determination

Finally, the instrument maximizing the a posteriori probability p(ωi | xk ) is determined as the identification result for thenote nk .

8

EURASIP Journal on Advances in Signal Processing
Table 3: Audio data on solo instruments.

Instr. no.

Name

01
09
15
31
33

Piano (PF)Classical guitar (CG)
Violin (VN)
Clarinet (CL)
Flute (FL)

Pitch range

Variation

A0–C8
E2–E5
G3–E7
D3–F6
C4–C7

1, 2, 3

Table 4: Instrument candidates for each part. The abbreviations of
instruments are defined in Table 3.
Part 1
Part 2
Part 3
Part 4

5.

PF, VN, FL
PF, CG, VN, CL
PF, CGPF, CG

EXPERIMENTS

5.2. Experiment 1: leave-one-out
The experiment was conducted using the leave-one-out
cross-validation method. When evaluating a musical piece,a mixed-sound template was constructed using the remaining two pieces. Because we evaluated three pieces, we constructed three diﬀerent mixed-sound templates by dropping
the piece used for testing. The mixed-sound templates wereconstructed from audio signals of solo and duo music (S+D)

Articulation

no. of data

Normal only

792
702
576
360
221



Forte, mezzo,
and piano




1, 2

Table 5: Number of notes in mixed-sound templates (Type I). Templates of Types II and III have about 1/2 and 1/3–1/4 times the notesof Type I (details are omitted due to a lack of space). S + D and
S + D + T stand for the templates constructed from audio signals ofsolo and duo music, and from those of solo, duo, and trio music,
respectively.
Number

Name

S+D

S+D+T

Subset∗No. 13

PF
CG
VN
CL
FL

31,33423,446
14,7607,332
4,581

83,491
56,18447,087
20,031
16,732

24,784
10,718
9,804
4,888
3,043

No. 16

PF
CG
VN
CL
FL

26,73819,76012,342
5,916
3,970

71,20346,924
39,461
16,043
14,287

21,104
8,893
8,230
3,944
2,632

No. 17

PF
CG
VN
CL
FL

23,836
17,618
11,706
5,928
3,613

63,93242,552
36,984
16,20813,05918,880
8,053
7,806
3,952
2,407

5.1. Data for experiments
We used audio signals generated by mixing audio data takenfrom a solo musical instrument sound database according
to standard MIDI files (SMFs) so that we would have correct data on F0s, onset times, and durations of all notes because the focus of our experiments was solely on evaluating
the performance of our instrument identification method byitself.
The SMFs we used in the experiments were three pieces
taken from RWC-MDB-C-2001 (Piece Nos. 13, 16, and 17)[29]. These are classical musical pieces consisting of four orfive simultaneous voices. We created SMFs of duo, trio, andquartet music by choosing two, three, and four simultaneous
voices from each piece. We also prepared solo-melody SMFs
for template construction.
As audio sources for generating audio signals of duo, trio,
and quartet music, an excerpt of RWC-MDB-I-2001 [30],listed in Table 3, was used. To avoid using the same audio
data for training and testing, we used 011PFNOM, 151VNNOM, 311CLNOM, and 331FLNOM for the test data andthe others in Table 3 for the training data. We prepared audio signals of all possible instrument combinations within
the restrictions in Table 4, which were defined by taking thepitch ranges of instruments into account. For example, 48
diﬀerent combinations were made for quartet music.

Dynamics∗

Template used in Experiment III.

and solo, duo, and trio music (S + D + T). For comparison,
we also constructed a template, called a solo-sound template,
only from solo musical sounds. The number of notes in eachtemplate is listed in Table 5. To evaluate the eﬀectiveness of
F0-dependent multivariate normal distributions and usingmusical context, we tested both cases with and without eachtechnique. We fed the correct data on the F0s, onset times,
and durations of all notes because our focus was on the performance of the instrument identification method alone.
The results are shown in Table 6. Each number in the table is the average of the recognition rates for the three pieces.
Using the DAMS method, the F0-dependent multivariate
normal distribution, and the musical context, we improved
the recognition rates from 50.9 to 84.1% for duo, from 46.1
to 77.6% for trio, and from 43.1 to 72.3% for quartet musicon average.
We confirmed the eﬀect of each of the DAMS method (mixed-sound template), the F0-dependent multivariate normal distribution, and the musical context using

Tetsuro Kitahara et al.

9

Table 6: Results of Experiment 1. : used, ×: not used; bold font denotes recognition rates of higher than 75%.
Template
F0-dependent
Context

Solo sound

S+D

S+D+T

×

×





×

×





×

×





×



×



×



×



×



×



PF
CG
VN
CL
FL

53.7%
46.0%
63.7%
62.9%
28.1%

63.0%
44.6%
81.3%
70.3%
33.5%

70.7%
50.8%
63.1%
53.4%
29.1%

84.7%
42.8%
75.6%
56.1%
38.7%

61.5%
50.9%
68.1%
81.8%
67.6%

63.8%
67.5%
85.5%
92.1%
84.9%

69.8%
70.2%
70.6%
81.9%
67.6%

78.9%
85.1%
87.7%
89.9%
78.8%

69.1%
44.0%
65.4%
84.6%
56.8%

70.8%
57.7%
84.2%
95.1%
70.5%

71.0%
71.0%
67.7%
82.9%
61.5%

82.7%
82.9%
88.1%
92.6%
74.3%

Av.

50.9%

58.5%

53.4%

59.6%

66.0%

78.8%

72.0%

84.1%

64.0%

75.7%

70.8%

84.1%

PF
CG
VN
CL
FL

42.8%
39.8%
61.4%
53.4%
33.0%

49.3%
39.1%
76.8%
55.7%
42.6%

63.0%
40.0%
62.2%
46.0%
36.7%

75.4%
31.7%
72.5%
43.9%
46.5%

44.1%
52.1%
67.0%
69.5%
68.4%

43.8%
66.8%
81.8%
77.1%
77.9%

57.0%
68.3%
70.8%
72.2%
68.1%

61.4%
82.0%
83.5%
78.3%
76.9%

52.4%
47.2%
60.5%
71.0%
59.1%

53.6%
62.8%
80.6%
82.8%
69.3%

61.5%
68.3%
68.1%
76.2%
64.0%

68.3%
82.8%
82.5%
82.8%
71.5%

Av.

46.1%

52.7%

49.6%

54.0%

60.2%

69.5%

67.3%

76.4%

58.0%

69.8%

67.6%

77.6%

PF
CG
Quartet VN
CL
FL

38.9%
34.3%
60.2%
45.8%
36.0%

46.0%
33.2%
74.3%
44.8%
50.8%

54.2%
35.3%
62.8%
39.5%
40.8%

64.9%
29.1%
73.1%
35.8%
52.0%

38.7%
51.2%
70.0%
62.6%
69.8%

38.6%
62.7%
81.2%
66.8%
76.1%

50.3%
64.8%
72.7%
65.4%
69.9%

53.1%
75.3%
82.3%
69.3%
76.2%

46.1%
51.2%
67.4%
68.6%
61.7%

46.6%
64.5%
79.2%
74.4%
69.4%

53.3%
65.0%
69.7%
70.9%
64.5%

57.2%
79.1%
79.9%
74.5%
70.9%

Av.

43.1%

49.8%

46.5%

51.0%

58.5%

65.1%

64.6%

71.2%

59.0%

66.8%

64.7%

72.3%

Corr.
347
19

S+D
Inc.
25
109

Duo

Trio

Table 7: Results of McNemar’s test for quartet music (Corr. = correct, Inc. = incorrect).
(a) Template comparison (with both F0-dependent and context)

S+D

Corr.
Inc.

Corr.
233
25

Solo sound
Inc.
133
109

χ02 = (133 − 25)2 /(133 + 25) = 73.82

S+D+T

Corr.
Inc.

Corr.
224
34

Solo sound
Inc.
148
94

χ02 = (148 − 34)2 /(148 + 34) = 71.40

S+D+T

Corr.
Inc.

χ02 = (25 − 19)2 /(25 + 19) = 1.5(b) With versus without F0-dependent (with S+
D + T template and context)(c) With versus without context (with S + D + T
template and F0-dependent model)

w/o F0-dpt.
Corr.
Inc.
314
58
25
103

w/o Context
Corr.
Inc.
308
64
27
101

w/ F0-dpt.

Corr.
Inc.

χ02 = (58 − 25)2 /(58 + 25) = 13.12

McNemar’s test. McNemar’s test is usable for testing whether
the proportions of A-labeled (“correct” in this case) data to
B-labeled (“incorrect”) data under two diﬀerent conditions
are significantly diﬀerent. Because the numbers of notes are
diﬀerent among instruments, we sampled 100 notes at random for each instrument to avoid the bias. The results of

w/ Context

Corr.
Inc.

χ02 = (64 − 27)2 /(64 + 27) = 15.04

McNemar’s test for the quartet music are listed in Table 7(those for the trio and duo music are omitted but are basically the same as those for the quartet), where the χ02 are test
statistics. Because the criterion region at α = 0.001 (which is
the level of significance) is (10.83, +∞), the diﬀerences except
for S + D versus S + D + T are significant at α = 0.001.

10
Other observations are summarized as follows.
(i) The results of the S+D and S+D+T templates were notsignificantly diﬀerent even if the test data were from quartet
music. This means that constructing a template from polyphonic sounds is eﬀective even if the sounds used for thetemplate construction do not have the same complexity as
the piece to be identified.
(ii) For PF and CG, the F0-dependent multivariate normal distribution was particularly eﬀective. This is because
these instruments have large pitch dependencies due to theirwide pitch ranges.
(iii) Using musical context improved recognition rates,on average, by approximately 10%. This is because, in themusical pieces used in our experiments, pitches in the
melodies of simultaneous voices rarely crossed.
(iv) When the solo-sound template was used, the use ofmusical context lowered recognition rates, especially for CL.
Because our method of using musical context calculates the apriori probability of each note on the basis of the a posteriori
probabilities of temporally neighboring notes, it requires anaccuracy suﬃcient for precalculating the a posteriori probabilities of the temporally neighboring notes. The lowered
recognition rates are because of the insuﬃcient accuracy ofthis precalculation. In fact, this phenomenon did not occur
when the mixed-sound templates, which improved the accuracies of the precalculations, were used. Therefore, musical context should be used together with some technique of
improving the pre-calculation accuracies, such as a mixedsound template.
(v) The recognition rate for PF was not high enough insome cases. This is because the timbre of PF is similar tothat of CG. In fact, even humans had diﬃculty distinguishing them in listening tests of sounds resynthesized from harmonic structures extracted from PF and CG tones.
5.3. Experiment 2: template construction fromonly one piece
Next, to compare template construction from only one piecewith that from two pieces (i.e., leave-one-out), we conducted an experiment on template construction from only
one piece. The results are shown in Table 8. Even when using
a template made from only one piece, we obtained comparatively high recognition rates for CG, VN, and CL. For FL, theresults of constructing a template from only one piece were
not high (e.g., 30–40%), but those from two pieces were close
to the results of the case where the same piece was used forboth template construction and testing. This means that avariety of influences of sounds overlapping was trained fromonly two pieces.
5.4. Experiment 3: insufficient instrument
combinations
We investigated the relationship between the coverage of instrument combinations in a template and the recognition
rate. When a template that does not cover instrument combinations is used, the recognition rate might decrease. If thisEURASIP Journal on Advances in Signal Processing
Table 8: Template construction from only one piece (Experiment2). Quartet only due to lack of space (unit: %).
S+D

S+D+T

16

17



16

17



PF (57.8)

32.3

38.4

36.6 (67.2)

33.2

45.1

39.7

CG (73.3)

78.1

76.2

76.7 (76.8)

84.3

80.3

82.1

13 VN (89.5)

59.4

87.5

86.2 (87.2)

58.0

85.2

83.1

CL (68.5)

70.8

62.2

73.8 (72.3)

72.3

68.6

75.9

FL (85.5)

40.2

74.9

82.7 (86.0)

38.9

68.8

80.8

PF

74.1

(64.8)

61.1

71.2

79.6

(67.1)

73.0

78.3

CG

79.2

(77.9)

78.9

74.3

70.4

(82.6)

74.0

75.2

16 VN

89.2

(85.5)

87.0

87.0

86.0

(83.5)

84.7

85.0

CL

68.1

(78.9)

68.9

76.1

72.4

(82.8)

76.3

82.1

FL

82.0

(75.9)

72.5

77.3

77.9

(72.3)

35.7

69.2

PF

53.0

39.4

(51.2) 51.6

52.2

40.6

(55.7) 53.7

CG

73.7

69.0

(75.8) 75.0

76.0

74.3

(78.4) 80.0

17 VN

79.5

61.2

(78.3) 73.6

77.4

58.0

(78.7) 71.7

CL

51.3

60.5

(57.1) 57.9

61.1

62.6

(66.9) 65.4

FL

65.0

35.0

(73.1) 68.7

58.6

34.7

(70.9) 62.6

13

13

 Leave-one-out. Numbers in left column denote piece numbers for test,
those in top row denote piece numbers for template construction.

Table 9: Instrument combinations in Experiment 3.
Solo

PF, CG, VN, CL, FL

Duo

PF–PF, CG–CG, VN–PF, CL–PF, FL–PF

Trio

Not used

QuartetNot useddecrease is large, the number of target instruments of thetemplate will be diﬃcult to increase because O(mn ) data areneeded for a full-combination template, where m and n are
the number of target instruments and simultaneous voices.
The purpose of this experiment is to check whether such a
decrease occurs in the use of a reduced-combination template. As the reduced-combination template, we used onethat contains the combinations listed in Table 9 only. These
combinations were chosen so that the order of the combinations was O(m). Similarly to Experiment 1, we used theleave-one-out cross-validation method. As we can see from
Table 10, we did not find significant diﬀerences between using the full instrument combinations and the reduced combinations. This was confirmed, as shown in Table 11, through
McNemar’s test, similarly to Experiment 1. Therefore, we expect that the number of target instruments can be increased
without the problem of combinational explosion.

Tetsuro Kitahara et al.

11

Trio

Quartet72.3 %
52.5 %

52.2 %

46.3 %
51 %

71.2 %

77.6 %

53.1%
75.3%
82.3%
69.3%
76.2%
71.2%

Quartet57.1 %

68.9%
52.4%
85.0%
71.1%
74.5%
70.4%

76.4 %

PF
CG
VN
CL
FL
Average60

56.3 %

78.9%
85.1%
87.7%
89.9%
78.8%
84.1%
61.4%
82.0%
83.5%
78.3%
76.9%
76.4%

48 %
54 %

85.4%
70.8%
88.2%
90.4%
79.7%
82.9%
73.9%
62.0%
85.7%
79.7%
76.5%
75.6%

60.7 %

PF
CG
VN
CL
FL
Average
PF
CG
VN
CL
FL
AverageTrio

84.1 % 84.1 %80

64.5 %

Full set

50.5 %
59.6 %

Duo

Subset

Duo

100

(%)

Table 10: Comparison of templates whose instrument combinations were reduced (subset) and not reduced (full set).

40
20
0

S

S+D S+D+T S

S+D S+D+T S

S+D S+D+T

PCA only
PCA+LDA

Figure 6: Comparison between using both PCA and LDA with using only PCA (Experiment 4). “Duo,” “trio,” and “quartet” represent
pieces for test (identification). “S,” “S+D,” and “S+D+T” represent
types of templates.

Table 12: ANOVA SS = sum of squares, DF = degrees of freedom,
DR = dimensionality reduction.
Table 11: Results of McNemar’s test for full-set and subset templates χ02 = (25 − 19)2 /(25 + 19) = 1.5.
SubsetFull set

Corr.
Corr. 341
19
Inc.

Inc.
25
115

5.5. Experiment 4: effectiveness of LDA
Finally, we compared the dimensionality reduction using
both PCA and LDA with that using only PCA to evaluate theeﬀectiveness of LDA. The experimental method was leaveone-out cross-validation. The results are shown in Figure 6.
The diﬀerence between the recognition rates of the solosound template and the S + D or S + D + T template was 20–
24% using PCA + LDA and 6–14% using PCA only. These
results mean that LDA (or DAMS) successfully obtained asubspace where the influence of the overlapping of soundsof multiple instruments was minimal by minimizing the ratio of the within-class variance to the between-class variance.
Under all conditions, using LDA was superior to not using
LDA.
We confirmed that combining LDA and the mixed-soundtemplate is eﬀective using two-way factorial analysis of variance (ANOVA) where the two factors are dimensionality reduction methods (PCA only and PCA + LDA) and templates(S, S + D, and S + D + T). Because we tested each condition
using duo, trio, and quartet versions of Piece Nos. 13, 16, and17, there are nine results for each cell of the two-factor ma-

Src. of var.

SSDF

F value

P valueDR
Template
Interaction
Residual
Total

0.336
0.302
0.057
0.158
0.855

1
2
2
48
53

102.08
45.75
8.75
—
—

1.806 × 10−13
7.57 × 10−12
5.73 × 10−4——

trix. The table of ANOVA is given in Table 12. From the table,
we can see that the interaction eﬀect as well as the eﬀects ofdimensionality reduction methods and templates are significant at α = 0.001. This result means that mixed-sound templates are particularly eﬀective when combined with LDA.
5.6.

Application to XML annotation

In this section, we show an example of XML annotation ofmusical audio signals using our instrument identification
method. We used a simplified version of MusicXML insteadof the original MusicXML format because our method does
not include rhythm recognition and hence cannot determine note values or measures. The document-type definition(DTD) of our simplified MusicXML is shown in Figure 7.
The main diﬀerences between it and the original one arethat elements related to notation, which cannot be estimatedfrom audio signals, are reduced and that time is represented
in seconds. The result of XML annotation of a piece of polyphonic music is shown in Figure 8. By using our instrument
identification method, we classified notes according to partand described the instrument for each part.

12

EURASIP Journal on Advances in Signal Processing<!ENTITY % score-header“(work?, movement-number?, movement-title?,
identification?, defaults?, credit∗,
part-list)”><!ELEMENT part-list (score-part+)><!ELEMENT score-part(identification?, part-name,part-abbreviation?, score-instrument)><!ATTLIST score-part
id ID #REQUIRED><!ELEMENT score-instrument(instrument-name, instrument-abbreviation?)><!ELEMENT instrument-name (#PCDATA)><!ELEMENT instrument-abbreviation (#PCDATA)><!ELEMENT score-partwise-simple>(%score-header;, part+)><!ATTLIST score-partwise-simple
version CDATA “1.0”><!ELEMENT part (note+)><!ATTLIST part
id IDREF #REQUIRED><!ELEMENT note (pitch, onset, oﬀset)><!ELEMENT pitch (step, alter?, octave)><!ELEMENT step (#PCDATA)><!ELEMENT alter (#PCDATA)><!ELEMENT octave (#PCDATA)><!ELEMENT onset (#PCDATA)><!ATTLIST onset
unit CDATA “sec”><!ELEMENT oﬀset (#PCDATA)><!ATTLIST oﬀset
unit CDATA “sec”>

Figure 7: DTD of our simplified MusicXML.

5.7. Discussion
We achieved average recognition rates of 84.1% for duo,
77.6% for trio, and 72.3% for quartet music chosen from fivediﬀerent instruments. We think that this performance is stateof the art, but we cannot directly compare these rates withexperimental results published by other researchers because
diﬀerent researchers used diﬀerent test data in general. We
also find the following two limitations in our evaluation:(1) the correct F0s are given;(2) nonrealistic music (i.e., music synthesized by mixing
isolated monophonic sound samples) is used.
First, in most existing studies, including ours, the methods were tested under the condition that the correct F0s
are manually fed [10, 13]. This is because the multiple<?xml version=“1.0” encoding=“UTF-8”
standalone=“no” ? ><!DOCTYPE score-partwise-simple SYSTEM“partwisesimple.dtd”><score-partwise-simple><part-list><score-part id=“P1”><part-name>Part 1</part-name><score-instrument>Piano</score-instrument></score-part><score-part id=“P2”><part-name>Part 3</part-name><score-instrument>Violin</score-instrument></score-part>······</part-list><part id=“P1”><note><pitch><step>G</step><alter>+1</alter><octave>3</octave></pitch><onset>1.0</onset><oﬀset>2.0</oﬀset></note><note><pitch><step>G</step><octave>3</octave></pitch><onset>2.0</onset><oﬀset>2.5</oﬀset></note><note><pitch><step>D</step><octave>4</octave></pitch><onset>2.5</onset><oﬀset>3.0</oﬀset></note>······</part><part id = “P2”><note><pitch><step>D</step><alter> +1 < /alter><octave> 4 < /octave></pitch><onset>1.5</onset><oﬀset> 2.488541 < /oﬀset></note><note><pitch><step>C</step><alter> +1 < /alter><octave> 4 < /octave></pitch><onset> 3.0 < /onset><oﬀset> 3.5 < /oﬀset></note>······</part>······</score-partwise-simple>
Figure 8: Example of MusicXML annotation.

Tetsuro Kitahara et al.
F0-estimation for a sound mixture is still a challenging problem, and the studies aimed at evaluating the performanceof only their instrument identification methods. If the estimated F0s are used instead of the manually given correct
F0s, the performance of instrument identification will decrease. In fact, Kinoshita et al. [11] reported that given random note patterns taken from three diﬀerent instruments,
the instrument identification performance was around 72–
81% for correct F0s but decreased to around 66–75% for estimated F0s. Because multiple-F0 estimation has actively beenstudied [8, 31, 32], we plan to integrate and evaluate our instrument identification method with such a multiple-F0 estimation method in the future.
Second, most existing studies, including ours, used nonrealistic music as test samples. For example, Kashino et al. [7]and Kinoshita et al. [11] tested their methods on polyphonicmusical audio signals that were synthesized by mixing isolated monophonic sounds of every target instrument on an
MIDI sampler. This was because information on the instrument for every note that was used as correct references inthe evaluation was then easy to prepare. Strictly speaking,however, the acoustical characteristics of real music are different from those of such synthesized music. The performance of our method would decrease for real music because
legato play sometimes causes overlapping successive notes
with unclear onsets in a melody and because sound mixtures often involve reverberations. We plan to manually annotate the correct F0 information for real music and evaluate
our method after integrating it with a multiple-F0 estimation
method as mentioned above.
6.

CONCLUSION

We have provided a new solution to an important problem
of instrument identification in polyphonic music: the overlapping of partials (harmonic components). Our solution
is to weight features based on their robustness to overlapping by collecting training data extracted from polyphonicsounds and applying LDA to them. Although the approachof collecting training data from polyphonic sounds is simple,
no previous studies have attempted it. One possible reason
may be that a tremendously large amount of data is required
to prepare a thorough training data set containing all possible sound combinations. From our experiments, however,
we found that a thorough training data set is not necessary
and that a data set extracted from a few musical pieces is
suﬃcient to improve the robustness of instrument identification in polyphonic music. Furthermore, we improved theperformance of the instrument identification using musicalcontext. Our method made it possible to avoid musically unnatural errors by taking the temporal continuity of melodies
into consideration.
Because the F0 and onset time of each note were given
in our experiments to check the performance of only the instrument identification, we plan to complete MusicXML annotation by integrating our method with a musical note estimation method. Our future work will also include the useof the description of musical instrument names identified us-

13
ing our method to build a music information retrieval systemthat enables users to search for polyphonic musical pieces bygiving a query including musical instrument names.
REFERENCES[1] M. Good, “MusicXML: an internet-friendly format for sheetmusic,” in Proceedings of the XML Conference & Exposition, Orlando, Fla, USA, December 2001.
[2] P. Bellini and P. Nesi, “WEDELMUSIC format: an XML music
notation format for emerging applications,” in Proceedings ofthe International Conference of Web Delivering of Music, pp. 79–
86, Florence, Italy, November 2001.
[3] B. S. Manjunath, P. Salembier, and T. Sikora, Introduction of
MPEG-7, John Wiley & Sons, New York, NY, USA, 2002.
[4] T. Nagatsuka, N. Saiwaki, H. Katayose, and S. Inokuchi, “Automatic transcription system for ensemble music,” in Proceedings of the International Symposium of Musical Acoustics (ISMA’92), pp. 79–82, Tokyo, Japan, 1992.
[5] G. J. Brown and M. Cooke, “Perceptual grouping of musicalsounds: a computational model,” Journal of New Music Research, vol. 23, pp. 107–132, 1994.
[6] K. D. Martin, “Automatic transcription of simple polyphonicmusic,” in Proceedings of 3rd Joint meeting of the Acoustical Society of America and Japan, Honolulu, Hawaii, USA, December
1996.
[7] K. Kashino, K. Nakadai, T. Kinoshita, and H. Tanaka, “Application of the Bayesian probability network to music scene analysis,” in Computational Auditory Scene Analysis, D. F. Rosenthaland H. G. Okuno, Eds., pp. 115–137, Lawrence Erlbaum Associates, Mahwah, NJ, USA, 1998.
[8] A. Klapuri, T. Virtanen, A. Eronen, and J. Seppanen, “Automatic transcription of musical recordings,” in Proceedings ofWorkshop on Consistent & Reliable Acoustic Cues (CRAC ’01),
Aalborg, Denmark, September 2001.
[9] Y. Sakuraba, T. Kitahara, and H. G. Okuno, “Comparing features for forming music streams in automatic music transcription,” in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP ’04), vol. 4, pp. 273–
276, Montreal, Quebec, Canada, May 2004.
[10] K. Kashino and H. Murase, “Sound source identification system for ensemble music based on template adaptation andmusic stream extraction,” Speech Communication, vol. 27,
no. 3, pp. 337–349, 1999.
[11] T. Kinoshita, S. Sakai, and H. Tanaka, “Musical sound source
identification based on frequency component adaptation,” inProceedings of IJCAI Workshop on Computational Auditory
Scene Analysis (IJCAI-CASA ’99), pp. 18–24, Stockholm, Sweden, July-August 1999.
[12] J. Eggink and G. J. Brown, “A missing feature approach to instrument identification in polyphonic music,” in Proceedingsof IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP ’03), vol. 5, pp. 553–556, Hong Kong,
April 2003.
[13] J. Eggink and G. J. Brown, “Application of missing feature theory to the recognition of musical instruments in polyphonicaudio,” in Proceedings of International Symposium on Music Information Retrieval (ISMIR ’03), Baltimore, Md, USA, October
2003.
[14] K. D. Martin, Sound-source recognition: a theory and computational model, Ph.D. thesis, MIT, Cambridge, Mass, USA, 1999.
[15] A. Eronen and A. Klapuri, “Musical instrument recognition using cepstral coeﬃcients and temporal features,” in14Proceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP ’00), vol. 2, pp. 753–756,
Istanbul, Turkey, June 2000.
[16] A. Fraser and I. Fujinaga, “Toward real-time recognition of
acoustic musical instruments,” in Proceedings of International
Computer Music Conference (ICMC ’99), pp. 175–177, Beijing,
China, October 1999.
[17] I. Fujinaga and K. MacMillan, “Realtime recognition of orchestral instruments,” in Proceedings of International Computer Music Conference (ICMC ’00), pp. 141–143, Berlin, Germany, August 2000.
[18] G. Agostini, M. Longari, and E. Pollastri, “Musical instrument
timbres classification with spectral features,” EURASIP Journal
on Applied Signal Processing, vol. 2003, no. 1, pp. 5–14, 2003.
[19] T. Kitahara, M. Goto, and H. G. Okuno, “Pitch-dependent
identification of musical instrument sounds,” Applied Intelligence, vol. 23, no. 3, pp. 267–275, 2005.
[20] J. Marques and P. J. Moreno, “A study of musical instrumentclassification using Gaussian mixture models and support vector machines,” CRL Technical Report Series CRL/4, Cambridge Research Laboratory, Cambridge, Mass, USA, 1999.
[21] J. C. Brown, “Computer identification of musical instruments
using pattern recognition with cepstral coeﬃcients as features,” Journal of the Acoustical Society of America, vol. 105,
no. 3, pp. 1933–1941, 1999.
[22] A. G. Krishna and T. V. Sreenivas, “Music instrument recognition: from isolated notes to solo phrases,” in Proceedings ofIEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP ’04), vol. 4, pp. 265–268, Montreal, Quebec, Canada, May 2004.
[23] B. Kostek, “Musical instrument classification and duet analysis
employing music information retrieval techniques,” Proceedings of the IEEE, vol. 92, no. 4, pp. 712–729, 2004.
[24] S. Essid, G. Richard, and B. David, “Instrument recognition inpolyphonic music,” in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP ’05),
vol. 3, pp. 245–248, Philadelphia, Pa, USA, March 2005.
[25] T. Kitahara, M. Goto, K. Komatani, T. Ogata, and H. G.
Okuno, “Instrument identification in polyphonic music: feature weighting with mixed sounds, pitch-dependent timbre
modeling, and use of musical context,” in Proceedings of 6th International Conference on Music Information Retrieval (ISMIR’05), pp. 558–563, London, UK, September 2005.
[26] A. S. Bregman, Auditory Scene Analysis: The Perceptual Organization of Sound, MIT Press, Cambridge, Mass, USA, 1990.
[27] D. Huron, “Tone and voice: a derivation of the rules of voiceleading from perceptual principles,” Music Perception, vol. 19,
no. 1, pp. 1–64, 2001.
[28] H. Kameoka, T. Nishimoto, and S. Sagayama, “Harmonictemporal-structured clustering via deterministic annealing
EM algorithm for audio feature extraction,” in Proceedings of
6th International Conference on Music Information Retrieval(ISMIR ’05), pp. 115–122, London, UK, September 2005.
[29] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC
music database: popular, classical, and jazz music databases,”in Proceedings of 3rd International Conference on Music Information Retrieval (ISMIR ’02), pp. 287–288, Paris, France, October 2002.
[30] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC
music database: music genre database and musical instrument
sound database,” in Proceedings of 4th International Conference on Music Information Retrieval (ISMIR ’03), pp. 229–230,
Washington, DC, USA, October 2003.

EURASIP Journal on Advances in Signal Processing[31] M. Goto, “A real-time music-scene-description system:
predominant-F0 estimation for detecting melody and basslines in real-world audio signals,” Speech Communication,
vol. 43, no. 4, pp. 311–329, 2004.
[32] H. Kameoka, T. Nishimoto, and S. Sagayama, “Audio stream
segregation of multi-pitch music signal based on time-spaceclustering using Gaussian Kernel 2-dimensional model,” inProceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP ’05), vol. 3, pp. 5–8,
Philadelphia, Pa, USA, March 2005.

Tetsuro Kitahara received the B.S. degreefrom Tokyo University of Science in 2002
and the M.S. degree from Kyoto University
in 2004. He is currently a Ph.D. Course Student at Graduate School of Informatics, Kyoto University. Since 2005, he has been a Research Fellow of the Japan Society for the
Promotion of Science. His research interests
include music informatics. He received fiveawards, including TELECOM System Technology Award for Student in 2004 and IPSJ 67th National Convention Best Paper Award for Young Researcher in 2005. He is aStudent Member of IPSJ, IEICE, JSAI, ASJ, JSMPC, and IEEE.
Masataka Goto received the Doctor ofEngineering degree in electronics, information, and communication engineeringfrom Waseda University, Japan, in 1998.
He then joined the Electrotechnical Laboratory (ETL), which was reorganized as the
National Institute of Advanced Industrial
Science and Technology (AIST) in 2001,
where he has been a Senior Research Scientist since 2005. He served concurrently asa Researcher in Precursory Research for Embryonic Science andTechnology (PRESTO), Japan Science and Technology Corporation(JST) from 2000 to 2003, and as an Associate Professor of the Department of Intelligent Interaction Technologies, Graduate Schoolof Systems and Information Engineering, University of Tsukuba,
since 2005. His research interests include music information processing and spoken language processing. He has received 18 awards,
including the Information Processing Society of Japan (IPSJ) Best
Paper Award and IPSJ Yamashita SIG Research Awards (special interest group on music and computer, and spoken language processing) from the IPSJ, the Awaya Prize for Outstanding Presentation
and Award for Outstanding Poster Presentation from the Acoustical Society of Japan (ASJ), Award for Best Presentation from theJapanese Society for Music Perception and Cognition (JSMPC),WISS 2000 Best Paper Award and Best Presentation Award, and Interaction 2003 Best Paper Award.
Kazunori Komatani is an Assistant Professor at the Graduate School of Informatics,
Kyoto University, Japan. He received a B.S.
degree in 1998, an M.S. degree in Informatics in 2000, and a Ph.D. degree in 2002, allfrom Kyoto University. He received the 2002
FIT Young Researcher Award and the 2004
IPSJ Yamashita SIG Research Award, bothfrom the Information Processing Society ofJapan.

Tetsuro Kitahara et al.
Tetsuya Ogata received the B.S., M.S., andPh.D. degrees of Engineering in mechanical engineering in 1993, 1995, and 2000,
respectively, from Waseda University. From1999 to 2001, he was a Research Associate
in Waseda University. From 2001 to 2003,
he was a Research Scientist in Brain Science
Institute, RIKEN. Since 2003, he has beena faculty member in Graduate School of Informatics, Kyoto University, where he is currently an Associate Professor. Since 2005, he has been a Visiting
Associate Professor of the Humanoid Robotics Institute of Waseda
University. His research interests include human-robot interaction,
dynamics of human-robot mutual adaptation, and intersensorytranslation in robot system. He received the JSME Medal for Outstanding Paper from the Japan Society of Mechanical Engineers in
2000.
Hiroshi G. Okuno received B.A. and Ph.D.
degrees from the University of Tokyo in
1972 and 1996, respectively. He worked for
Nippon Telegraph and Telephone, JST Kitano Symbiotic Systems Project, and Tokyo
University of Science. He is currently a Professor of Department of Intelligence Technology and Science, Graduate School of Informatics, Kyoto University. He was a Visiting Scholar at Stanford University, and Visiting Associate Professor at the University of Tokyo. He has done
research in programming languages, parallel processing, and reasoning mechanism in AI, and he is currently engaged in computational auditory scene analysis, music scene analysis, and robot audition. He received various awards including the 1990 Best Paper
Award of JSAI, the Best Paper Award of IEA/AIE-2001 and 2005,
and IEEE/RSJ Nakamura Award for IROS-2001 Best Paper Nomination Finalist. He was also awarded 2003 Funai Information Science Achievement Award. He edited with David Rosenthal “Computational Auditory Scene Analysis” from Lawrence Erlbaum Associates in 1998 and with Taiichi Yuasa “Advanced Lisp Technology”
from Taylor and Francis Inc. in 2002. He is a Member of the IPSJ,
JSAI, JSSST, JCCS, RSJ, ACM, IEEE, AAAI, ASA, and ISCA.

15