Musical Sound Source Identi cation Based on
Frequency Component Adaptation
Tomoyoshi Kinoshita and Shuichi Sakai and Hidehiko Tanaka
Graduate School of Information Engineering, The University of Tokyo
7-3-1 Hongo Bunkyo-ku, Tokyo, 113-8656, JAPANE-mail: {kino,sakai,tanaka}@mtl.t.u-tokyo.ac.jp
TEL: +81-3-5841-7413, FAX: +81-3-5800-6922
Abstract
In auditory scene analysis, sound source identication is an essential operation when extracting musical notes from acoustical signals composed of multiple sound sources. We have previously proposed a processing model OPTIMA
for music scene analysis and implemented its
experimental system. However, the system was
not robust to signals with overlapped frequency
components. In this paper, we present a new
method that improves this problem by using
overlap pattern of frequency components, andimplemented as a processing module in OPTIMA. Weighted template-matching method isapplied to identify sound sources repeatedly toeach frequency component cluster. The weightis evaluated according to the signi cance ofeach feature of the signal. When multiple components are overlapped, our system adaptively
modi es features of an input signal to a combination of overlapped components. Experimental results show that the system can identify
sound sources of 66% to 75% of musical notes.
It also showed about 10% improvement in accuracy, compared to the result without the proposed mechanism.

1 Introduction
As we can recognize the surrounding scene from acoustic signals received through our ears, human being andother animals are endowed with this function from birth.
By contrast, it is still unable for computers or other machinery to realize their acoustic environment. To builda system that works in the real world, realization of thisfunction is a large and essential step. In the eld ofauditory psychology, Bregman's work [Bregman, 1990]
proposed some basic theory on this function | auditory
scene analysis |. However, its realization on a computer is not examined in the work, and has attracted
much interest of researchers.
Accordingly, several works have been done on auditory scene analysis by computers. Lesser et al. presented

IPUS [Lesser et al., 1995] based on blackboard architecture. In this system, agents work following a rule-based
policy, thus exibility and scalability is not achieved somuch. Nakatani et al. employed a residue-driven architecture for sound segregation [Nakatani et al., 1995]. The
system has multiple agents, and two agents called tracer
and eraser extract acoustic signal from one source. Ellis
proposed a system based on a prediction-driven method[Ellis, 1996]. This system is sensitive to the context of
the input stream. Our current method does not copewith context, but after note hypotheses are obtained, information integration mechanism works to re ect otherclues including scene context.
All of these works segregate streams in acoustic signals
and do not identify their sound source. As a system that
identi es sound sources, Kashino et al.'s work, named
OPTIMA [Kashino et al., 1995] is the basis of the proposed system. OPTIMA has multiple processing modules that work independently. After the processes, eachresult is expressed as probabilistic information and integrated into the nal result. In OPTIMA, sound source
identi cation is performed in a similar way as the current
system. It also uses physical features of frequency components, such as power ratio of harmonics and sharpnessof onset. However, identi cation is done in a simple way
based on principle component analysis and discriminant
analysis. Even if multiple components overlap and features are a ected, OPTIMA processes them as they are.
Therefore, OPTIMA cannot identify sound sources withoverlapped components, and this problem has waited forsolutions.
Another Kashino's work [Kashino and Murase, 1997]
uses wave-form template, and through template lteringand phase tracking, the template is arranged according
to the input signal. Generally, signals from two individual instruments vary signi cantly even if both are
the same kind of sound source. Kashino pointed thatdivergent phase between frequency components causes
this phenomena, and introduced a phase tracking mechanism. However, when a template was not prepared aswave-form data but frequency components, this problem
would be solved, since time-frequency analysis generally
discards phase information of each component.
In this paper, we propose a novel system for soundInput Acoustic Signal

Source
Piano
Piano
..
.
Clarinet..
.

PreproccessFrequency Components

Sound Formation

Overlap Patterns

Component Clusters

Feature
#1
0.724
0.271
..
.
0.513
..
.

Feature
#2
23.48
18.22
..
.
49.11
..
.

Feature
#3
5.901
3.725
..
.
7.224
..
.

111
111
111
111
111
..

.

Table 1: Example of a feature template.

Feature Extraction
FeaturesFeature Adaptation

Frequency Components

Adapted Features

Frequency

Matching
Piano
Similarities
Time

Hypotheses Creation
Source Nameof one Cluster

Clarinet

Total Result
ViolinPostprocess

Frequency

Frequency

Frequency...
Notes, Score or MIDI data

Feature Template
Knowledgebase
Time

Figure 1: Con guration of proposed system.

Cluster 1

Time

Cluster 2

Clusters

source identi cation. Our system uses an adaptive
method for component's feature analysis and achievedmuch better identi cation accuracy than OPTIMA.
The system architecture is described in Section 2. And
experimental results are shown in Section 3. Finally,
conclusion is described in Section 4.

2 System Overview
The system has seven processing blocks and aknowledge-base of frequency components' features ofsound sources. In this paper, we call this knowledgebase, a feature template. Figure 1 shows the overall
diagram.
Frequency components are extracted from input
acoustic signal in the Preprocesses stage. In theSound Formation stage, components obtained in theprevious stage are clusterized, where each cluster corresponds to a musical note. Overlap pattern is also extracted in this stage. The system computes featuresfrom each component in the Feature Extraction stage.
Extracted features are adapted in the Feature Adaptation stage in accordance with the overlap pattern. In
the Matching stage, the adapted features and templates
in Feature Template Knowledge-base are compared andsimilarity is evaluated. The similarities are applied to
the Hypotheses Creation stage, where note hypotheses are created. When all the clusters' sources are not
identi ed, the process is restored to the Feature Adap-

Time

Shared by Cluster 1, 2

Overlap PatternFigure 2: Clusterize into notes and obtain overlap pattern.
tation stage. Finally, note data or scores are createdin the Postprocesses stage. Details of each stage aredescribed in the following subsections.
Feature Template has a set of records of features.
Each record consists of a sound source name and a listof feature values (Table 1).
2.1

Preprocesses and Sound Formation

The system rst performs time frequency analysis to obtain a sound spectrogram from the input and tone model
signals. Then frequency components are extracted. In
this process, we chose the IIR lter-bank method andthe pinching planes method[Kashino et al., 1995].
In the second stage, sound formation is performed.
Through this process, frequency components are clusterized into musical notes.
We used Kashino's
method[Kashino et al., 1995] for the task. In thismethod, distortion of harmony and common onset between two frequency components are extracted, and clustering is performed using the result. Additionally, we extracted patterns of frequency components' overlap. Here\overlap pattern" is de ned as a set of combinations ofcomponents shared by multiple clusters (Figure 2).

 Power of each component
 Center frequency (power-weighted average of
frequency)

Here, center frequency is de ned as follows:
X

pf

.X

Categorize into feature groupspwhere p and f is the power and frequency of eachfrequency component in a power-spectrogram.
 Skewness and kurtosis of each component's powerenvelope
Here, skewness is de ned as follows:

FragilefeatureAdditivefeaturePerform
adaptation

Preferential
featureOverlap pattern

E [(p 0 )3 ]=3

Overlap pattern

Invalidand kurtosis is de ned as follows:

Overlap pattern

Invalid

Collect adapted features

E [(p 0 )4 ]= 4 0 3

where p is the power at each sample point of frequency components and  is the mean of p.

Invalid
Invalid

Table 2: List of frequency component features (excerpt).
2.2

Figure 3: Adaptation mechanism.

Feature Extraction

Next, the system extracts features of frequency components. Here, feature is de ned as physical parameterssuch as a shape of power envelope, a strength of an attack, or a power ratio of each component. Table 2 shows
the list of the adopted features.
When the power of some harmonics is too small, features extracted from it is invalidated and would not be
used to obtain similarity in the Matching stage.
2.3

To Matching stage

Feature Adaptation

In real music, multiple notes are usually presented simultaneously and several frequency components are overlapped. This causes transformation of frequency components and their features. Therefore, we cannot immediately use the extracted features.
In this stage, features are arranged in accordance withthe features' characteristics and the components' overlap
pattern (Figure 3).
Categorize Features

We categorized features by their characteristics into
three types as listed below:
1. Additive features
When a frequency component overlaps with others,
its feature is generally the sum of each component's
feature.
(eg. power of component)
2. Preferential features
The feature becomes the maximum or minimum
value of overlapping components.
(eg. strength of attack)3. Fragile features
If multiple components overlap, the feature value is
not signi cant any longer.
(eg. skewness of component's envelope)
Adaptation

When multiple components overlap, the system recalculates each feature according to the feature's type, following the algorithms described below. In case a component
is not shared by other clusters, nothing is done.
1. Additive features
Adaptation is performed in the following algorithm:
If the sound source of a cluster which shares thiscomponent is already determined
Then

Feature value is considered to be the sum ofmultiple feature values, and is subtracted by
the mean of the sharing component's feature
value in the template.

Else

The system cannot estimate the added feature
value. Therefore, nothing is done.

2. Preferential features
Adaptation is performed in the following algorithm:
If the sound source of a cluster which shares thiscomponent is already determinedThen

If feature value is close to the sound source's

feature value in the template

Then

The feature value is considered to belong to
the processing note, and nothing is done.

Case 1: Large weightElse

The feature value is considered not to belong
to the processing note, and the component isinvalidated.

Feature value

Else

The system cannot decide whether the feature
value belongs to the processing note or not.
Therefore, nothing is done.

Distriburion of source A

Distriburion of source B

Case 2: Small weight

3. Fragile features
In this case, the feature value is considered to becorrupted, and the component is invalidated.
If sound sources of some components are already determined, the recalculation is performed applying theirfeature templates.
2.4

Case1:

Matching

Source identi cation is performed by weighted-matching
method between sound source feature templates andadapted features.
Weight calculation

The system computes the mean and standard deviationfrom feature templates of each source in advance.
Then, the weight of each feature is obtained by thefollowing equation; in this paper, we use superscript as
the feature id and subscript as the sound source id:

jX j  j 0  j



w

i
s;t

Feature value

=P

i
t

i
s



(1)

i
s

Case2:

If the feature value distributions of two soundsources are suciently apart, the feature isconsidered appropriate as a clue to identify
sources.
When distributions are close, the feature isconsidered inappropriate.

Figure 4: Weighting based on the distribution of feature
values.
Before the similarity evaluation, distance between thetarget input signal and features in the template databaseis calculated. The distance is obtained as follows:
i

s

W =
i
s

1

X

jS j 0 1 2 6= w
t

S;t

i
s;t

2

(2)

s

where S denotes the set of sound sources (=fPiano,
Clarinet, 1 1 1g) and s denotes each sound source (s 2 S ).
P is aRprobabiluty
p value2in normal distribution (P (jX j zz) = 0z (1= 2) exp(0x =2)dx).
Here, we de ne Dsi as the distribution of the s-thsource's i-th feature. Using the equation above, Wsi
becomes larger when Dsi is isolated from other soundsources. For example, when Dsi is suciently apart fromother sources' Dsi , the weight Wsi becomes 1, and when
all Dsi have equivalent mean values, the weight becomes
0. Therefore, the i-th feature value with large Wsi is assumed to be signi cant in the s-th source's identi cation(Figure 4). Wsi is applied to similarity evaluation in thenext stage.
Similarity Calculation

The system evaluates the similarity between features of
the input signal and each template. The similarity represents the con dence of source identi cation.

!

f 0
d = P jX j 

i
s

i
s

(3)

i
s

where dis , f i , it and ti denote the distance of the i-thfeature between the input and the s-th source's template,the i-th feature value of the input signal, average andstandard deviation of the i-th feature of the s-th source's
template, respectively. P is a probability function aspreviously denoted.
Then, the similarity is calculated by the following
equation:S = exp

Xsi

W log d
i
s

i
s

,
X

!

W

i
s

(4)

i

where Wsi and dis are obtained beforehand.
In this equation, i represents each feature, but whena feature value is invalidated, the summation is skipped.
The following list describes the case when a feature becomes invalid:
1. A frequency component is not extracted because its
power is too small. The feature value is invalidated
during the Frequency Component Extraction stage.

Source
Name
Piano
Clarinet
ViolinPower

Feature Name
temporal symmetry of rst harmonics
temporal symmetry of third harmonics
skewness of rst harmonics
kurtosis of rst harmonics
temporal symmetry of third harmonics
power of second harmonics
attack strength of rst harmonics
attack strength of second harmonics
kurtosis of rst harmonics

Frequency

Piano

Time
Power

Frequency

Clarinet

Table 3: Top three large weighted features.
Time

2. A component overlaps with another one and the feature is fragile.
2.5

After the matching process is performed, the system xes
the source name of the note with the lowest pitch among
the clusters, and feeds it back to the Feature Adaptation
stage. In the Feature Adaptation stage, adaptation is
performed again. At this time, one cluster's source nameis determined and its template is used in calculation.
When all clusters' sources are identi ed, the systemcreates note hypotheses. Each hypothesis has multiple
notes, and each note has metrics such as onset time,
duration time, pitch and source name.
2.6

Postprocesses

In OPTIMA architecture, note hypotheses obtained in
the last stage and other probabilistic information are integrated afterwards[Kashino et al., 1995]. Through thisprocess, errors included in the hypotheses are expected
to be corrected.

3 Evaluation

We evaluated the proposed system in two ways. First,
feature's weight value is calculated and its validity isevaluated. Next, source identi cation accuracy for random note pattern is evaluated. In this paper, the e ectiveness of the proposed method is proved by comparing
results of recognition with and without feature adaptation and weight calculation.
3.1

Power

Frequency

ViolinHypotheses Creation

Weight Calculation

First, Table 3 shows the experimental result of Wsi calculation in the Matching Stage.
This result matches our intuition quite well. Piano's
power has a sharp attack and a calm decline, and mostof its power is distributed at the beginning of the envelope. Low value of Temporal symmetry or large skewness
re ects asymmetry of Piano's power envelope. Clarinet
has little power in even harmonics and each component
has trapezoidal power distribution. Therefore, kurtosis
and power of the second harmonics becomes lower thanother sources, and temporal symmetry becomes larger.

Time

Figure 5: Typical shapes of sound source's frequency
component.
Violin's onset is relatively gentle in the sources we used.
Hence attack strength is a signi cant clue to identify
whether a source is a violin or not. (Figure 5)
3.2

Experiment to Random Note Pattern

We prepared a random note pattern in advance to theevaluation. The random note pattern consists of pairsof notes. The sound source and pitch of each note arechosen at random. Random notes are categorized into
three classes:
Class 1: The second harmonics of a note overlaps withthe base component of another note. In this class,all components of the note overlaps with components of other notes. This is the most dicult caseamong the three classes.
Class 2: The second harmonics of a note and the third
harmonics of another note overlap. These two notes
have an interval of perfect fth; frequencies of thenotes' base components are 2 : 3.
Class 3: Patterns categorized into neither Class 1 nor
2. Overlap of components rarely occurs.
Here, each class has 300 patterns, and a total of 900
note patterns were processed in the system. In this experiment, we used signal from di erent instruments for
input and feature templates.
Figure 6 shows the result of the note creation. This is
the recognition accuracy when source name misrecognition is ignored, which shows the upper-boundary of thesource identi cation accuracy.

%RWK&ODVV

1RQH:LWKRXWDGDSWDWLRQ:LWKRXWZHLJKW%RWK&ODVV





&ODVV



&ODVV





&ODVV

&ODVV














Figure 6: Result of note creation for random note patterns.
Figure 7 shows the source identi cation accuracy.
These bars indicate the percentage of correctly-identi ed
notes. Each bar shows the result when feature adaptation and weight calculation are performed or not, as
follows:
Bar Title
WeightFeatureCalculation Adaptation
Both
Performed Performed
Without adaptation
Without weight

Performed





Not
performed
Performed

Not
performed
None
Not
Not
performed performed
Note that accuracy stands for the average of recall andprecision.
This result includes failures in the note creation stage,
and does not show independent accuracies of source identi cation. Figure 8 shows the result when the pitch ofeach note is given before source identi cation is performed.

4 Conclusions
We have presented a new method for musical soundsource identi cation, which enables the identi cation forcomponent-overlapped signals by feature adaptation.
The experimental result showed the e ectiveness of
this method. In the case of random note pattern, therecognition accuracy has improved from 70% to 81% in
the most e ective case; Class 2.
In real music, multiple notes have frequencies of integral ratio to express ne harmony, and componentoverlaps like Classes 1 and 2 in this paper appear often.
Therefore, the proposed method is e ective also to thesource identi cation of musical signals.
For practical use, the accuracy of the proposed system
is insucient and more improvement is waited for. In the











Figure 7: Result of whole process for random note patterns.
1RQH:LWKRXWDGDSWDWLRQ:LWKRXWZHLJKW%RWK&ODVV






&ODVV




&ODVV















Figure 8: Result of source identi cation when the pitch
is given beforehand.
adaptation stage, we used a simple algorithm for overlapped components, such as \Do nothing" and \Mark as
invalid". We are going to examine more sophisticatedways which consider other informations like the di erence between types of instruments.
We used data from di erent individual of the same
instrument for input and feature templates of the experiment. Most of the in uence is avoided through thecalculation of mean of feature value, but in some cases
the system failed to identify the source name because ofthis in uence. Therefore, this problem should be solved.
Some errors appeared in the evaluation result arecaused by component extraction failure. Its improvement is also our future work.

Acknowledgments

This research is supported by Grant-in-Aid for JSPS Fellows (No.09-07628) and JSPS Fellowships for Young Scientists.
We also would like to thank NTT Communication
Science Laboratories for the permission to use theNTTMSA-P1 acoustic signal database.

References[Bregman, 1990] A. S. Bregman. Auditory scene analysis. MIT Press, 1990.
[Ellis, 1996] Daniel P. W. Ellis. Prediction-driven Computational Auditory Scene Analysis. PhD thesis, Department of Electrical Engineering and Computer Science, M.I.T., 1996.
[Kashino and Murase, 1997] Kunio Kashino and Hiroshi
Murase. A music stream segregation system based on
adaptive multi-agents. In Proceedings of the 15th International Joint Conference on Arti cial Intelligence,
volume 2, pages 1126{1131, August 1997.
[Kashino et al., 1995] Kunio
Kashino,Kazuhiro Nakadai, Tomoyoshi Kinoshita, and Hidehiko Tanaka. Organization of hierarchical perceptual
sounds. In Proceedings of the 14th International Joint
Conference on Arti cial Intelligence, Volume 1, pages
158{164. Morgan Kaufmann Publishers, Inc., August
1995.
[Lesser et al., 1995] Victor R. Lesser, S. Hamid Nawab,
and Frank I. Klassner. IPUS: An architecture for the
integrated processing and understanding of signals.
Arti cial Intelligence, 77:129{171, 1995.
[Nakatani et al., 1995] Tomohiro Nakatani, Hiroshi G.
Okuno, and Takeshi Kawabata. Residue-driven architecture for computational auditory scene analysis.
In Proceedings of the 14th International Joint Conference on Arti cial Intelligence, volume 1, pages 165{
172, August 1995.