PAPERS

In Search of a Perceptual Metric for Timbre:
Dissimilarity Judgments among Synthetic Soundswith MFCC-Derived Spectral Envelopes
HIROKO TERASAWA,1,2 AES Member , JONATHAN BERGER3, AND SHOJI MAKINO1(terasawa@tara.tsukuba.ac.jp)
1(brg@ccrma.stanford.edu)(maki@tara.tsukuba.ac.jp)

Life Science Center of TARA, University of Tsukuba 1-1-1 Tennodai, Tsukuba, Ibaraki 305-8577, Japan
2
JST, PRESTO (Information Science and Humans) 7 Gobancho, Chiyoda-ku, Tokyo 102-0076, Japan
3
CCRMA, Department of Music, Stanford University 660 Lomita Drive, Stanford, CA 94305, USA

This paper presents a quantitative metric to describe the multidimensionality of spectralenvelope perception, that is, the perception specifically related to the spectral element of timbre. Mel-cepstrum (Mel-frequency cepstral coefficients or MFCCs) is chosen as a hypothetical
metric for spectral envelope perception due to its desirable properties of linearity, orthogonality,
and multidimensionality. The experimental results confirmed the relevance of Mel-cepstrum
to the perceived timbre dissimilarity when the spectral envelopes of complex-tone synthetic
sounds were systematically controlled. The first experiment measured the perceived dissimilarity when the stimuli were synthesized by varying only a single coefficient from MFCC.
Linear regression analysis proved that each of the 12 MFCCs has a linear correlation with
spectral envelope perception. The second experiment measured the perceived dissimilarity
when the stimuli were synthesized by varying two of the MFCCs. Multiple regression analysis
showed that the perceived dissimilarity can be explained in terms of the Euclidean distance ofthe MFCC values of the synthetic sounds. The quantitative and perceptual relevance between
the MFCCs and spectral centroids is also discussed. These results suggest that MFCCs canbe a metric representation of spectral envelope perception, where each of its orthogonal basis
functions provides a linear match with human perception.

0 INTRODUCTION
The spectral envelope of a sound is a crucial aspect oftimbre perception. In this study, we propose a quantitativemodel of spectral envelope perception, that is, the spectral
element in the timbre perception, with a set of orthogonal basis functions. The goal of this work is to develop
a quantitative mapping between a physical description of
the spectral envelope and its perception, with the purposeof controlling timbre in sonification in a meaningful andreliable way. The model suggests a systematic descriptionof spectral envelope perception whose simplicity may beseen as analogous to the three primary colors in the visualsystem.
In the earliest studies of timbre perception, Helmholtz
speculated that the spectral envelope is the source of thetimbre variations [1]. For speech sounds, the formant structure of the overtone series was determined to be the key factor in differentiating vowels [2], [3]. For Western musicalinstrument sounds, timbre perception has often been described in terms of the spectral centroid, spectral flux, and674

attack time [4]–[7]. In addition to these factors, other factors
such as amplitude and frequency micromodulations and inharmonicity are also taken into account [8]. Although thesedescriptive studies can address the relationship between
the physical aspects of sound and the perception, more information on the precise shape of the spectral envelope isoften needed to synthesize sounds in a controlled way. In
other words, although there are multiple layers (i.e., perceptual, cognitive, physical, and social perspectives) in addressing sound quality [9], understanding at one layer does
not necessarily lead to the improvement at another layer.
Recent studies on morphed instrumental sounds employed
the time-varying multiband approach to evaluate the perception of the synthesized timbre, connecting these multiple
layers [10]–[12].
A robust quantitative model for timbre perception has
been long desired for the control of timbre in sound synthesis, especially in relation to the use of sound in auditory displays of information. To take full advantage of themultidimensionality of timbre in sonification, we need aquantitative, multidimensional description for spectral
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

PAPERSenvelope perception. Such a model allows reliable mappings of data to perceptual space, which is critical for effective sonification [13]. Many researchers have conceptualized spectral envelope perception by analogy with thevisual color system, by finding an orthogonal basis in the
spectral shapes of instrumental sounds [14], by proposing the concept of sound color [15], and by visualizingorgan sounds as an energy-balance transition across threefrequency regions [16].
In this work, we aim for a simple, quantitative, and multidimensional model that can be extended to synthesize
perceptually meaningful variations of spectral envelopes.
Ideally, such a model will predict the spectral envelope
perception in a linear and orthogonal manner; each orthogonal basis should have a quantitative label that can linearly
represent the perceived difference, and the perception of acomplex spectral envelope could be explained in terms of
the superposition of these basis functions.
Seeking such a model for spectral envelope perception,
we chose the Mel-cepstrum (also known as Mel-frequency
cepstrum coefficients or MFCCs) for the following reasons: (1) MFCCs are constructed by a set of orthogonalbasis functions, therefore satisfying the need for an orthogonal model; (2) MFCCs are based on perceptually relevant
scalings, which can provide a linear mapping between the
numeric description and the perception; and (3) MFCCs
have been a powerful front-end tool for many engineering
applications, and clarifying the perceptual characteristicsof MFCCs by performing psychoacoustic experiments isvaluable.
The Mel-cepstrum was originally proposed as “the description of short-term spectra ... in terms of the contribution
to the spectrum of each of an orthogonal set of spectrumshape functions” [17]. The Mel-cepstrum is computed byapplying a discrete cosine transform (DCT) to the output
of a simple auditory filterbank that roughly resembles critical bands. Unlike other representations of spectral envelope, such as the 1/3-octave-band models or specific loudness, the basis functions of a Mel-cepstrum are mathematically orthogonal. Mermelstein noted that a Mel-cepstrum
can constitute a distance metric that reflects the perceptual space of phonemes [18] and examined its efficiency
as a front end for automatic speech recognition [19]. Now
it is considered to be the classic front-end algorithm for
automatic speech recognition [20]. Its application has been
extended to timbre-related music information retrieval [21],[22], sound database indexing based on timbre characteristics [23], [24], timbre control for sonification [25], perceptual description of instrumental sound morphing [26], anda proposal that timbre perception be represented in termsof sound color and sound density [27].
Despite such numerous applications, the authors’ earlier
works were the first to examine the Mel-cepstrum’s perceptual characteristics with psychoacoustic experiment procedures [28]–[30], and, before that, the perceptual relevanceof MFCCs was demonstrated only by applications. Therefore, it is worthwhile to examining the perceptual characteristics of MFCCs in detail using psychoacoustic experiments. Still, Mel-cepstrum is not the most precise auditory
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

IN SEARCH OF A PERCEPTUAL METRIC FOR TIMBRE

model. Other perceptual models, such as specific loudness[31], the spatiotemporal receptive-field model [32], andthe Mellin transform [33] may seem to be better options.
However, these models do not consist of orthogonal basis
functions, and they are not necessarily a compact algorithm
that enables efficient analysis and synthesis of timbre. For
these reasons, MFCCs were considered the most suitable
for a spectral envelope perception model.
We employed the following framework to test this model.
We first synthesized a stimulus set with gradually changing
spectral envelopes by varying the Mel-cepstrum values in a
stepwise order, while keeping the temporal characteristics
constant across the stimuli. The participants listened to thestimuli in pairs and provided dissimilarity ratings. Finally,
the relationship between the dissimilarity ratings and the
Euclidean distance of the MFCC values was analyzed witha linear regression.
To measure spectral envelope perception, the temporalcharacteristics of the stimuli must be strongly controlled
because the temporal structure has a strong effect on timbre
perception. To control this effect, we decided to use thesame temporal structure for all of the stimuli. Although it
might seem more interesting to employ various kinds of
temporal structures in a single experiment, it would notallow us to observe the multidimensionality of spectral envelope perception accurately. In musical instrument timbre
studies, Plomp detected three dimensions for spectral envelope perception when he minimized the variation in thetemporal structure [14], whereas other researchers detected
only a single dimension (spectral centroid) dedicated solely
to the spectral envelope, in addition to another spectrotemporal dimension (spectral flux) when they introduced
various temporal structures [4]–[7]. Therefore, we decided
to maintain a single kind of temporal structure for the entire
stimuli set.
In designing the temporal structure of the stimuli, we
wanted to create tones with a distinct quality that helped
the participants make reliable judgments. For this purpose,the stimuli are desirably sustained and have the fewest random factors. The simplest design that satisfies this criterion is obviously the addition of sinusoids in a harmonicseries. But this design has an unwanted effect: when the
spectral envelope is manipulated, the amplified partials
are perceived as obtrusive and separated from the otherpartials.
To avoid this perceptual segregation, we added a vibratolike frequency modulation to all the harmonics, so that allof the partials contribute to a unified tone thanks to the“common fate” effect [34]. With this vibrato, the synthesized sounds exhibited a voice-like quality that is morenatural than sinusoid beeps. Because parameter-mapping
sonification can sound unpleasant [35], such naturalness
is valuable. As already shown in voice-based sonification
projects, voice-like qualities often facilitate the comprehension of data [36], [37]. However, stimuli with vibrato
may be unacceptable for the experiment because vibrato
might influence spectral envelope perception due to its dramatic musical effect, which is particular to Western operatic
singing. But, in fact, adding vibrato to a voice does not
675

TERASAWA ET AL.

PAPERS
0.015

0.01
amplitudechange the perceived vowel [38], and people can distinguish subtle changes in the spectral envelope of the tones
with vibrato [39]. This means that adding vibrato does notinterfere with the perception of the spectral envelope andthat, therefore, the use of vibrato for the experiment stimuli is acceptable. Furthermore, we expect that the inclusionof vibrato implies a musical setting and encourages theparticipants to engage in “musical listening” with greaterattention to timbre.
Using these stimuli, we conducted two experiments inthe experimental framework described above: the first was
designed to test the perceptual effect when modifying asingle dimension from MFCC, and the second to test theorthogonality of the timbre space using two dimensions
from MFCC. We used linear regression to analyze our data
because we were explicitly investigating the relationshipbetween MFCC and subjective ratings, rather than exploring unknown dimensions that could be discovered with themultidimensional scaling (MDS) method.
This paper aims to show (1) that there is a linear relationship between each of the Mel-cepstrum orthogonal
functions and the perceived timbre dissimilarity, (2) that themultidimensionality of complex spectral envelope perception can be explained in terms of the Euclidean distance ofthe orthogonal function coefficients, and (3) that the widelyused Mel-cepstrum can form a valid representation of spectral envelope perception. However, the multidimensionalityof spectral envelope perception beyond two dimensions andthe temporal aspect of timbre perception remain outside thescope of this study.
In the following sections, we describe the method weused to synthesize the stimuli while varying the MFCC
values in a controlled way. We describe our two experimentson spectral envelope perception and their result followed bya discussion and our conclusion.

0.005

0

0.2

domain, and provides the filterbank output, Fi :
 fi·highFi =Hi ( f ) · S( f ) df,⎧⎨200.0
Bandwidth (Hi ) = 133.3⎩1000 · 1.072i−13(i = 1)(1 < i ≤ 13).
(i > 13)(1)

The filterbank, whose triangular frequency response isshown in Fig. 1, is applied to the sound in the frequency
676

(2)

f = f i·low

where i is the channel number in the filterbank, f is thefrequency, Hi (f) is the filter response of the ith channel, andS(f) is the absolute value of the discrete Fourier transformof a signal. fi·low and fi·high denote the lowest and highest frequency bins, respectively, of the passband of the ith channel
filter.
The MFCCs, Ci , are computed by taking the DCT of thelog-scaled filterbank output:
L i = log10 (Fi ),Ii=1

1.1 Mel-Cepstrum
The MFCC is the DCT of a modified spectrum, in which
its frequency and amplitude are scaled logarithmically. Of
the various implementations that exist, the Mel-cepstrum
algorithm from Auditory Toolbox [40] was employed. The
spectrum is first processed with a filterbank of 32 channels, which roughly approximate the spacing and bandwidth of the auditory system’s critical bands. The frequency
response of the filterbank Hi (f) is shown in Fig. 1, and the
passband of each triangular window Hi (f) is shown in Eq.
(1). The amplitude of each filter is normalized so that eachchannel has unit power gain.

2

Fig. 1. Frequency response of the filterbank used for the MFCC.
The sound spectrum is first processed with this filterbank, whichroughly approximates the characteristics of auditory criticalbands. Taking the lower coefficients from the DCT of this filterbank output yields MFCC.

C n = wn
1 MFCC-BASED SOUND SYNTHESIS

0.5
1frequency (kHz)(3)

L i · cosπ(2i − 1) · (n − 1)
,
2I

(4)

√
√
where w0 = 1/ I , wn = 2/I for 1 ≤ n ≤ N − 1. I and N
represent the total number of filters and the total number ofMel-cepstrum coefficients, respectively. Taking 13 lower
coefficients from Cn , the set of coefficients from C0 to
C12 is called the MFCC which summarizes the spectralenvelope.
1.2 Sound Synthesis
The sound synthesis for the stimuli has two stages: (1)the spectral envelope is created by the pseudo-inverse transform of the Mel-cepstrum, and (2) an additive synthesis ofsinusoids is performed using the spectral envelope generated earlier.

1.2.1 Pseudo-Inversion of MFCC
As described above, the MFCC takes only the 13 lower
coefficients, and therefore it is a lossy transform from aspectrum. The inversion of the MFCC is not possible ina strict sense. This section describes the pseudo-inversion
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

PAPERSIN SEARCH OF A PERCEPTUAL METRIC FOR TIMBRE

of the MFCC, which generates a smooth spectral envelope
from a given Mel-cepstrum.
The generation of the spectral envelope starts with agiven array of Mel-cepstrum coefficients Cn , which is anarray of 13 coefficients. The reconstruction of the spectral
shape from the MFCC starts with the inverse discrete cosine
transform (IDCT) and amplitude scaling:L̃ i =

N

wn · Cn · cos

n=1π(2i − 1) · (n − 1),2IF̃i = 10 L̃ i .

(5)
(6)

In this pseudo-inversion, the reconstructed filterbank output F̃i is considered to represent the value of the reconstructed spectral envelope S̃( f ) at the center frequency ofeach channel from the filter bank,S̃( f i ) = F̃i ,(7)where fi is the center frequency of the ith auditory filter.
Therefore, to obtain a reconstruction of the entire spectrum, S̃( f ), a linear interpolation was applied to the values
between the center frequencies S̃( f i ).

1.2.2 Additive Synthesis
The voice-like stimuli used in this study are synthesizedusing additive sinusoidal synthesis. The reconstructed spectral envelope S̃( f ) determines the amplitude of each sinusoid. A slight amount of vibrato is added to give somecoherence and life to the resulting sound.
In the synthesis, a harmonic series is prepared, and thelevel of each harmonic is weighted based on the desired
smooth spectral shape. The pitch, or fundamental frequency
f0 , is set at 200 Hz, with the frequency of the vibrato v0 set
at 4 Hz and the sampling rate at 8 kHz.
Using the reconstructed spectral shape S̃( f ), the additive
synthesis of the sound is accomplished as follows:s(t) =QS̃( f inst (q, t)) · sin(2πq f 0 t + 1 − q cos 2πv0 t),

q=1

(8)
where q specifies the qth harmonic of the harmonic series. The total number of harmonics Q is 19, and all theharmonics stay under the Nyquist frequency of 4 kHz. The
amplitude of each harmonic is determined by using a lookup
table of S̃( f ) and the instantaneous frequency finst , which
is defined as follows:f inst (q, t) = q f 0 + qv0 · sin 2πv0 t.

2 EXPERIMENT 1: SPECTRAL ENVELOPEPERCEPTION OF SINGLE-DIMENSIONAL MFCC
FUNCTION
2.1 ScopeThis experiment considers the linear relationship between spectral envelope perception and each coefficientfrom the Mel-cepstrum, namely, a single function from theorthogonal set of spectral envelope functions. Following thesound-synthesis method described in the previous section,
when a coefficient from Mel-cepstrum changes graduallyin a linear manner while the other coefficients are kept constant, the spectral envelope of the resulting sound holds asimilar overall shape, but the humps of the envelope change
their amplitudes exponentially. In the experiment, it was examined whether the Mel-cepstrum can linearly represent the
spectral envelope perception, and all 12 coefficients from
Mel-cepstrum were tested based on this framework. The
experiment was granted the approval for human-subject
research by the Stanford University Institutional Review
Board.

2.2 Method
2.2.1 Participants
Twenty-five participants (graduate students and staff
members from the Center for Computer Research in Music and Acoustics at Stanford University) volunteered forthe experiment. The participants were aged 20–35 years
old, and had a musical background (majoring or minoringin music in college and graduate school), and/or an audio
engineering background (enrolled in a music technology
degree program). They all described themselves as having
normal hearing. We conducted a pilot study with Japaneseengineering students, and confirmed that the experimental results did not depend significantly on the participantgroup.

(9)

The fundamental frequency f0 = 200 (Hz) is determinedfrom the range of 180–230 Hz (the fundamental frequencyof the female voice), so that the MFCC of the resulting
sound maintains the intended stepwise or grid structure thebest.
The duration of the resulting sound s is 0.75 s. For the first
30 ms of the sound, its amplitude is linearly fading in, andfor the last 30 ms, its amplitude is linearly fading out. All
the stimuli are scaled with an identical scaling coefficient.
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

The specific loudness [31] of all the stimuli showed a verysmall variance, and their loudness was considered to be
fairly similar within the stimuli set. For all of the 144 stimuli
synthesized for this study, 123 stimuli scored under 3%, 10
stimuli scored 3–6%, and 7 stimuli scored 6–8% loudness
deviations when compared with the mean loudness of all
the stimuli.

2.2.2 Stimuli
Twelve sets of synthesized sounds were prepared. The set
n is associated with the MFCC coefficient Cn , the stimuli set
1 consists of the stimuli with C1 varied, and the stimuli set 2
consists of the stimuli with C2 varied, and so on. Although
Cn is increased from zero to one with five levels, namely,
Cn = 0, 0.25, 0.5, 0.75, 1.0, to form a stepwise structure,
the other coefficients are kept constant, that is, C0 = 1 andall the other coefficients are set at zero.
677

Amplitude

Amplitude

PAPERS

2

2

2

2

2

1

1

1

1

1

1

0.95
0
0
2

2

4

1
0
0
2

2

4

1
2

4

1
0
0
2

0
0
2

0
0
2

4

0
0
2

2

4

1
2

4

1
2

0
0
2

0
0
2

4

0
0
2

2

4

1
2

4

1
2

0
0
2

0
0
2

4

0
0
2

2

4

0
0
2

2

4

0
0
2

1

1

1

1

0
0

0
4 0

0
4 0

0
4 0

0
4 0

2
kHz

2
kHz

4

0.9

2

4

2
kHz

2

4

2
kHz

4

For example, stimuli set 4 consists of five stimuli basedon the following parameter arrangement:(10)

where C4 is varied with five levels:
C4 = [0, 0.25, 0.5, 0.75, 1.0].

(11)

Fig. 2 illustrates the idea of varying a single coefficientof MFCC, and the resulting set of the spectral envelopes
for the cases of varying C1 ,C2 , C3 , and C6 .

2.2.3 Procedure
The experiment had 12 sections, one for each of the 12
sets of stimuli. Each section consisted of a practice phaseand an experimental phase.
The task of the participants was to listen to a pair of stimuli that were played in sequence with a short intervening
silence, and to rate the perceived timbre dissimilarity of thepresented pair. They rated the perceived dissimilarity on ascale of 0 to 10, with 0 indicating that the presented pair ofsounds were identical, and 10 indicating that they were themost different within the section.
The participants pressed the “Play” button of the experiment GUI to play a sound, and reported the dissimilarity
rating using a slider on the GUI. To facilitate the judgment,
the pair with the largest spectral envelope difference in thesection (i.e., the pair of stimuli with the lowest and highest,
Cn = 0 and Cn = 1, is assumed to have a perceived dissimilarity of 10) was presented as a reference pair throughout
the practice and experimental phases. Participants were allowed to listen to the test pair and the reference pair as
many times as they wanted, but were advised not to repeat
678

0.75

0.7

Fig. 2. Spectral envelopes generated by varying a single Melcepstrum coefficient. The first row shows the spectral envelopes
when C1 from MFCC was varied from 0 to 1 with five steps(0, 0.25, 0.5, 0.75, and 1.0). The second, third, and fourth rows
correspond, respectively, to cases where C2 , C3 , and C6 from
MFCC were varied in the same manner.

C = [1, 0, 0, 0, C4 , 0, ..., 0],

0.85

0.8

1

1
2
kHz

2

1

1
2

0
0
2

R−squared

Amplitude

Amplitude

TERASAWA ET AL.

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 All

Fig. 3. Coefficients of determination (R2 ) from the linear regression analysis of Experiment 1 with 95 % confidence intervals foreach of the 12 Mel-cepstrum coefficients, Cn , and for the averageof all the coefficients.

this too many times before making their final decision on
scaling and proceeding to the next pair.
In the practice phase, five sample pairs were presented
for rating. In the experimental phase, 25 pairs per section(all the possible pairs from five stimuli) were presented ina random order. The order of presenting the sections was
also randomized. The participants were allowed to take abreak as they wished.

2.3 Linear Regression Analysis
The dissimilarity judgments were analyzed using simple linear regression [41], with absolute Cn differences as
the independent variable, and their reported perceived dissimilarities as the dependent variable. The coefficient ofdetermination R2 represents the goodness of fit in the linear regression analysis. The linear regression analysis was
individually applied for each section and each participant,because it is anticipated that every listener could responddifferently to the stimuli sets, which would result in thedeviation of the regression coefficients. With a quantile–
quantile plot, the R2 values formed a straight line except
for a very few outliers with low R2 values, showing that the
distribution of the R2 values is close to normal.
After the linear regression, the R2 values for one section
from all the participants were averaged to find the meandegree of fit (mean R2 ) of each section. The mean R2 among
the participants was used to judge the linear relationshipbetween the Cn distance and the perceived dissimilarity.
The mean R2 and the corresponding confidence interval
are plotted in Fig. 3. The mean R2 for all the responses was
85%, with the confidence intervals for all the sections overlapped. This means that all of the coefficients, from C1 to
C12 , have a linear correlation with the perception of soundcolor with a statistically equivalent degree of fit, when an
experiment is performed on an individual coefficient independent of other coefficients.
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

PAPERSIN SEARCH OF A PERCEPTUAL METRIC FOR TIMBRE

3.2 Method
3.2.1 Participants
Nineteen participants, who were audio engineers, administrative staff members, visiting composers, and artists
from the Banff Centre, Alberta, Canada, volunteered forthis experiment. The participants were aged 25–40 years
old, and they had a strong interest in music, with many ofthem having received professional training in music and/oraudio engineering. All of them described themselves asnormal-hearing.

3.2.2 Stimuli
Five sets of synthesized sounds were prepared that wereassociated with the five different kinds of two-dimensional
subspaces. The five subspaces were made by varying [C1 ,C3 ], [C3 , C4 ], [C3 , C6 ], [C3 , C12 ], and [C11 , C12 ], respectively. For each set, the coefficients in question were independently varied over four levels (Cn = 0, 0.25, 0.5, 0.75,
and Cm = 0, 0.25, 0.5, 0.75) to form a grid-like structure;
the other coefficients were kept constant, that is, C0 = 1
and all other coefficients were set at zero. By varying two
coefficients independently, over four levels, each set had 16
synthesized sounds.
For example, the first set made of the subspace [C1 , C3 ]consists of the 16 sounds based on the following parameterarrangement:
C = [1, C1 , 0, C3 , 0, ..., 0],(12)where C1 and C3 were varied over four levels, creating agrid with two variables.
The subspaces were chosen with the intention of testing
the spaces made of: nonadjacent low to middle coefficients([C1 , C3 ] and [C3 , C6 ]); two adjacent low coefficients ([C3 ,C4 ]); low and high coefficients ([C3 , C12 ]); and two adjacenthigh coefficients ([C11 , C12 ]).
Fig. 4 shows an example of the generated spectralenvelopes for this experiment.
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

Amplitude
Amplitude

3.1 ScopeThis experiment tested the spectral envelope perceptionof the two-dimensional MFCC subspace. The stimuli set
was synthesized by varying two coefficients from the Melcepstrum, say Cn and Cm , to form a two-dimensional subspace. The subjective response to the stimuli set was tested
based on the Euclidean space hypothesis, namely, that eachcoefficient functions as an orthogonal basis when estimating the spectral envelope perception. As it is not realistic to test all of the 144 two-dimensional subspaces, five
two-dimensional subspaces were chosen for testing. The
experiment was approved for human subject research by
the Stanford University Institutional Review Board.

Amplitude

Amplitude

3 EXPERIMENT 2: SPECTRAL ENVELOPEPERCEPTION OF TWO-DIMENSIONAL MFCC
SUBSPACE

2

2

2

2

1

1

1

1

0
0
2

2

4

1
0
0
2

2

4

4

0
0
2

2

4

0
0
2

2

4

4

0
0

2

4

0
0
2

2

4

0
0
2

2

4

4

0
0

2

4

0
0
2

2

4

2

4

2
kHz

4

1
2

4

1
2
kHz

0
0
2
1

1

1
2
kHz

0
0
2
1

1

1
0
0

2

1

1
0
0
2

0
0
2

0
0
2
1

2
kHz

4

0
0

Fig. 4. Spectral envelopes generated by varying two Mel-cepstrum
coefficients. The horizontal direction (left to right) corresponds toincrementing C6 from 0 to 0.75 in four steps (0, 0.25, 0.5, and0.75), and the vertical direction (top to bottom) corresponds toincrementing C3 from 0 to 0.75 in four steps. For example, thetop-left subplot shows the spectral envelope when C6 = C3 = 0,
and the bottom-right subplot is when C6 = C3 = 0.75.

3.2.3 Procedure
There are 16 stimuli sounds per one subspace, making
256 possible stimulus pairs. Because testing all the pairs
would take too much time and exhaust the participants, it
was necessary to reduce the number of the stimulus pairs
in the experiment. The strategies for reducing the test pairs
were (1) test either AB or BA ordering when measuring theperceived difference of stimuli A and B, instead of measuring the perception for both AB and BA; and (2) test
only some interesting pairs instead of testing all the possible combinations of stimulus pairs. We adopted the firststrategy, and the actual order for a stimulus pair in the experiment was randomly selected from AB and BA ordering.
However, the selection of ordering for each stimulus pair
was not varied across the participants.
To employ the first strategy, it was necessary to evaluate
whether the ordering of the stimuli had a significant effecton the perceived dissimilarity of the spectral envelope. To
compare the AB responses and BA responses, equivalence
testing was conducted based on confidence intervals [42].
First, regression analyses with AB order and BA order wereseparately conducted for each section and each participant.
Then the difference between the R2 values of AB and BA
order regressions for each section was calculated. After that,for each section, the mean and the confidence intervals for
the R2 differences were calculated across participants. The
confidence intervals of the differences for each section were
2–3.5%, falling into the predefined 5% minimum difference
range. This reveals that the regression analyses based on AB
responses and BA responses were statistically equivalent.
Because of this equivalency, it was decided that presenting
only one of two possible directions of a stimulus pair was
sufficient.
679

TERASAWA ET AL.

PAPERS1R−squared

0.9

Fig. 5. Selection of the test pairs for the two-dimensional MFCC
subspace experiment. Left: 16 pairs to examine distances from theorigin. Middle: 5 pairs to examine large distances. Right: 13 pairs
to examine some shorter parallel and symmetric distances.

0.8

0.7

0.6

0.5

Even after halving the number of stimulus pairs, there
were still too many and further reduction was needed.
Therefore, some pairs were chosen to represent large andsmall distances with some geometric order in the parametersubspace. Within each subspace, the test pairs were selectedwith the following interests, resulting in the total of 34 test
pairs per section:• From the zero of the space Cn = Cm = 0 to all thenodal points of the grid on the parameter subspace(16 pairs);• Other large distances (5 pairs);• Some shorter parallel and symmetric distances totest if they have similar perceived dissimilarities (13pairs).
The final configuration of the test pairs is presented inFig. 5 .
The participants’ task was to listen to the paired stimuli,
which were played in sequence with a short intervening
silence, and to rate the perceived timbre dissimilarity of thepresented pair using a 0 to 10 scale. Here 0 indicates thatthe paired stimuli were identical, and 10 indicates that theperceived dissimilarity between the paired stimuli was thelargest in the section.
The participants reported the dissimilarity rating using aslider on the experiment’s GUI. To facilitate the judgment,
the pair with the greatest spectral envelope difference inthe section is presented as a reference pair throughout thepractice and experimental phases, assuming that the pair ofstimuli with the lowest and highest, Cn = Cm = 0 and Cn =Cm = 0.75, would have a perceived dissimilarity of 10
within the stimuli set. Participants were allowed to listen
to the test pair and the reference pair as many times as
they wanted, but they were advised not to repeat this too
many times before making their final decision on scaling
and proceeding to the next pair.
In the practice phase, five sample pairs were presented
for rating. In the experimental phase, 34 pairs per section
were presented in a random order. The order of presenting
the sections was also randomized. The participants wereallowed to take breaks as they wished.
680

Sec 1

Sec 2

Sec 3

Sec 4

Sec 5

Fig. 6. Coefficient of determination (R2 ) from the regression analysis of the two-dimensional sound color experiment with 95%
confidence interval. Sections 1–5 represent the tests on subspaces[C1 , C3 ], [C3 , C4 ], [C3 , C6 ], [C3 , C12 ], and [C11 , C12 ], respectively.

3.3 Linear Regression Analysis
The dissimilarity judgments were analyzed using linear
regression. The orthogonality of the two-dimensional subspaces was tested with a Euclidean distance-based model:
the independent variable is the Euclidean distance of the
MFCC between the paired stimuli, and the dependent variable is the subjective dissimilarity rating:d 2 = ax 2 + by 2 ,(13)where d is the perceptual distance that subjects reported in
the experiment, x and y are the respective differences between the Cn and Cm values of the paired stimuli. This model
reflects the idea that the perceptual distance should be described in terms of the Euclidean distance of the spectralenvelope description vectors. The standard least-squares
estimation is used with the linear regression analysis. The
coefficient of determination, R2 , represents the goodness of
fit in the linear regression analysis.
Individual linear regression for each section and eachparticipant was applied first, and the R2 values of one section from all the participants were then averaged to find themean degree of fit (mean R2 ) of each section. The mean
R2 among the participants is used to determine whether theperceived dissimilarity reflects the Euclidean space model.
The mean R2 and the corresponding 95% confidence
interval are plotted in Fig. 6. The mean R2 of all the responses was 74% with the confidence intervals for all thesections overlapping. This means that all of the five subspaces demonstrate a similar degree of fit to a Euclidean
model of two-dimensional sound color perception regardless of the various choices of coordinates from the MFCC
space.
Fig. 7 shows the regression coefficients [i.e., a and b from
Eq. (13)] for each of the two variables from the regression
analysis for all five sections. The mean regression coefficients were consistently higher for the lower one of the two
MFCC variables, which means that lower Mel-cepstrum
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

PAPERSIN SEARCH OF A PERCEPTUAL METRIC FOR TIMBRE

Regression Coeff.

150

100

50

0

C1 C3

C3 C4

C3 C6

C3 C12

C11 C12Fig. 7. Regression coefficients from regression analysis of thetwo-dimensional sound color experiment. The first two points on
the left represent the regression coefficients for each dimensionof the [C1 , C3 ] subspace, followed by the regression coefficientsfor the subspaces of [C3 , C4 ], [C3 , C6 ], [C3 , C12 ], and [C11 , C12 ].

coefficients are perceptually more significant. Although theconfidence intervals overlap for the lower-order MFCCs,
and not for the higher-order MFCCs, this trend as regards
the mean regression coefficients is consistent across all the
MFCC subspace arrangements. This can be interpreted asindicating that the degree of contribution of the MFCCs
is similar in the low- to mid-order MFCCs with a slightly
decreasing trend, and for higher-order MFCCs, the degreeof contribution drops more quickly and significantly.

4 DISCUSSION
4.1 Representing the Spectral Envelope
Perception with MFCC
This section integrates the two experiments and discusses
whether an MFCC can be a fair representation for spectralenvelope perception. To summarize Experiment 1, it was
shown that every orthogonal basis from the MFCC is linearly correlated to spectral envelope perception with anaverage degree of fit of 85%. This holds true for every single coefficient from the 12 dimensions in the MFCC vector,
meaning that each of the coefficients is directly associatedwith spectral envelope perception. Experiment 2 tested theassociation between spectral envelope perception and twodimensional MFCC subspace. The Euclidean distance in
the MFCC explains the spectral envelope perception withan average degree of fit of 74%. Five different arrangementsof two-dimensional subspaces were selected, and all the arrangements showed a similar degree of fit to the Euclidean
distance model. An examination of the regression coefficients demonstrated that lower MFCC coefficients had astronger effect in the perceived sound color space. Thesefindings suggest that the MFCC can satisfy the desired
characteristics of the spectral envelope perception model
described in “Introduction”.
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

The limitation of this experiment is that it only measured
the responses to single-dimensional and two-dimensional
MFCC subspaces. However, for further dimensionality,
Beauchamp reported that the full dimensional MFCC canrepresent the timbre perception of musical instrument
sounds with a comparable precision to the Mel-band or
harmonics-based representations [43]. Other successful applications such as automatic speech recognition [20] or music information retrieval [21] suggest that the MFCC can efficiently retrieve timbre-related information such as vowels,consonants, and types of musical instruments. The recent
work by Alluri and Toviainen reports that the polyphonictimbre of excerpts from musical works may not be necessarily well described using an MFCC[44]. However, because
the scope of this experiment was the perception of musically organized mixtures of complex instrumental sounds,
this finding does not deny the capability of the MFCC torepresent the spectral envelope perception.
Previous works and applications have demonstrated that
the MFCC is a useful description for timbre-related information, but did not show how each of the MFCC components contributes to the overall performance of the whole
MFCC system. The experiments in this study showed thateach of the coefficients linearly correlates to the spectralenvelope perception and that there is a linear mapping between the perceived dissimilarity of the spectral envelope
and the Euclidean distance in a two-dimensional MFCC
subspace. These findings, along with Beauchamp’s fulldimensional MFCC study, suggest that the MFCC can bea fair representation of spectral envelope perception, andthat spectral envelope perception can be fully described interms of the Euclidean space constituted by MFCCs.
4.2 Associating the Spectral Centroid and an
MFCCThis section discusses the relationship between an MFCC
and the spectral centroid in representing the spectral envelope perception. A spectral centroid has a clear, strong correlation with the perceived brightness of sound [45], which
is an important factor in timbre perception [6].
First, to compare the spectral centroid with the MFCC,
the linear regression analysis of Experiment 1 was conducted using the spectral centroid of stimuli as an independent variable. The results were almost identical and statistically equivalent to Fig. 3. To investigate this effect, thespectral centroid for each of the stimuli used in Experiment
1 was calculated, which is shown in Fig. 8. This illustrates
that when a single dimension of the MFCC is manipulated, the resulting stimuli have a linear increase/decreasein the spectral centroid. The C1 stimuli had lower centroids while C1 was increasing from 0 to 1, and the C2
stimuli had higher centroids while C2 was increasing, butwith a smaller coefficient (less slope), and so on. In summary, lower MFCC coefficients have a stronger correlation
to the spectral centroid, and the correlation is negative for
odd-numbered MFCC dimensions (the spectral centroid decreases while Cn increases, where n is an odd number), andpositive for even-numbered MFCC dimensions (the spectral
681

TERASAWA ET AL.

PAPERS

2600

2400
C4 = 0C4 = 0.25
C4 = 0.5
C4 = 0.75

C2

2500

2350
C4

2400

C8
C10

2300

C12
C11
C9
C7

2200
C5

2100
C3

spectral centroid (Hz)

spectral centroid (Hz)

C6

2300

2250

2200

2000

2150

1900
C1

1800
0

0.2

0.4
0.6
MFCC value

0.8

1

2100
0

0.25

0.5

0.75

C3 valueFig. 8. Spectral centroid of the stimuli used for Experiment 1,
when a single coefficient from the Mel-cepstrum was varied from0 to 1 in five steps.

centroid increases while Cn increases, where n is an evennumber).
This is not a coincidence based on the trend in spectralenvelopes generated for this experiment as shown in Fig. 2.
The spectral envelopes generated by varying C1 have ahump around the low-frequency range, which corresponds
to the cosine wave at ω = 0, and a dip around the Nyquist
frequency, which corresponds to ω = π/2. As C1 increases,
the magnitude of the hump becomes higher.
The concentrated energy around the low-frequency region corresponds to the fact that the spectral centroids arelower while the value of C1 increases. Now, if the spectralenvelopes are generated by varying C2 , there are two humps
at the lowest frequency and the Nyquist frequency that correspond to ω = 0 and ω = π. Another hump at the Nyquist
frequency makes the spectral centroid higher, whereas increasing the value of C2 increases the spectral centroid.
The same trends are conserved for odd- and even-numbered
MFCC coefficients. With higher orders of MFCC, the basis
function has its humps more sparsely distributed over the
spectrum, which results in a weaker correlation between
the MFCC and the spectral centroid (i.e., the slope of theline in Fig. 8 becomes more shallow as n increases).
Furthermore, the results from Experiment 2 show thatthe lower-order Mel-cepstrum coefficient is perceptuallymore important. As shown in Fig. 9, the linear relationship between the MFCC and spectral centroid is consistentin the stimuli set for Experiment 2. The low coefficient’s
strong association with the spectral centroid can explain
this effect. Because of the correlation between the spectral
centroid and MFCC in the stimuli for Experiment 2, theresult of the regression analysis based on the spectral centroid was very similar to Fig. 6, except for Section 1. For
Section 1, the R2 of the spectral-centroid-based regression
was 84%, scoring it 13% above the R2 of the MFCC-based
regression, without overlapping confidence intervals. This
682

Fig. 9. Spectral centroid of the stimuli used for Experiment 2,
Section 2, when two coefficients from the Mel-cepstrum, C3 andC4 , were varied from 0 to 0.75 in four steps.

could be explained in terms of the coefficient choice ofC1 and C3 , which have a strong correlation with the spectral centroid in the same direction, and therefore are easily
confused. For Sections 2–5, the R2 of the MFCC-based regression was consistently higher by 2–5% than the R2 ofspectral-centroid-based regression, with overlapping confidence intervals.
The above-mentioned characteristics can be dependent
on the specific MFCC implementation, and the pseudoinversion of the MFCC used in this experiment. Depending
on how the MFCC and its inversion are implemented, it
could have different kinds of relationships to the spectral
centroid. The relevance between the MFCC and spectral
centroid present in this experiment may be generalizedwith further mathematical rationalization. If it is mathematically promised that higher Mel-cepstrum coefficients
have a weaker correlation with the spectral centroid resulting in the reduced perceptual significance, it may explain
the efficiency of the common practice, which uses only
12 or 13 lower coefficients from the MFCC for automatic
speech recognition or music information retrieval.
However, there was a trend in the spectral centroids in
the MFCC-based stimuli set for both experiments, and our
results do not conflict with the previously reported characteristics of the spectral centroid in relation to the timbre perception. Both Experiments 1 and 2 suggest that an
MFCC-based description holds a similar degree of linearity
in predicting spectral envelope perception to a spectralcentroid-based description. Yet the spectral centroid isessentially a single-dimensional descriptor and does notdescribe the complex shapes of the spectral envelope itself. Two sounds with different spectral envelopes could
have the same spectral-centroid value, but be represented
with different Mel-cepstrum values. The multidimensional
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

PAPERS

Mel-cepstrum delivers more information about the spectralenvelope than the spectral centroid.

IN SEARCH OF A PERCEPTUAL METRIC FOR TIMBRE

preparation of this article. We also thank Jim Beauchamp,
Andrew Horner, Michael Hall, and Tony Stockman for theirhelpful comments. This work was supported by France–
Stanford Center for Interdisciplinary Studies, The Banff
Centre, AES Educational Foundation, and JST-PRESTO.

5 CONCLUSION
On the basis of desirable properties for modelingspectral envelope perception (linearity, orthogonality, andmultidimensionality), Mel-frequency cepstral coefficients(MFCCs) were chosen as a hypothetical metric for modeling spectral envelope perception. Quantitative data fromtwo experiments illustrate the linear relationship between
the subjective perception of vowel-like synthetic sounds
and the MFCC.
The first experiment tested the linear mapping between
spectral envelope perception and all 12 Mel-cepstrum coefficients. Each Mel-cepstrum coefficient showed a linear
relationship to the subjective judgment at a statisticallyequivalent level to any other coefficient. On average, the
MFCC explains 85% of spectral envelope perception whena single coefficient from the MFCC is varied in an isolatedmanner from all the other coefficients.
In the second experiment, two Mel-cepstrum coefficientswere simultaneously varied to form a stimulus set in a twodimensional MFCC subspace, and the relevant spectral envelope perception was tested. A total of five subspaces were
tested, and all five exhibited a linear relationship between
the perceived dissimilarity and the Euclidean distance ofthe MFCC at a statistically equivalent level. A subjective
dissimilarity rating showed an average correlation of 74%
with the Euclidean distance between the Mel-cepstrum
coefficients of the tested stimulus pair. In addition, theobservation of regression coefficients demonstrated thatlower-order Mel-cepstrum coefficients influence spectralenvelope perception more strongly.
The use of MFCCs to describe spectral envelope perception was further discussed. Such a representation canbe useful not only in analyzing audio signals, but also incontrolling the timbre in synthesized sounds. The correlation between the MFCC and the spectral centroid was
also discussed, although such a correlation can be specificto our experimental conditions, and further mathematicalinvestigation is needed.
These experiments examined the MFCC model at low
dimensionality. Much work remains to be done in understanding how MFCC variation across the entire 12 dimensions might relate to human sound perception. An interesting approach is currently being employed by Horner andcoworkers, who are taking their previous experimental dataon timbre morphing of instrumental sounds [10, 11] and reanalyzing it using MFCC [26], [43]. Their approach using
instrumental sounds will provide a good complement to theapproach taken here.
6 ACKNOWLEDGMENT
We thank Malcolm Slaney for his contributions in establishing this research, and for his generous support in the
J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

7 REFERENCES[1] H. Helmholtz, On the Sensation of Tone (translation
by Alexander John Ellis), pp. 64–65 (Dover Publications,Mineola, NY, Original German Edition in 1863, English
translation in 1954).
[2] J. B. Allen, “How do humans process and recognize speech?,” IEEE Trans. Speech Audio Process., vol. 2,
pp. 567–577 (1994 Oct.).
[3] G. E. Peterson and H. L. Barney, “Control methods
used in a study of the vowels,” J Acoust Soc Am., vol. 24,
no. 2, pp. 175–184 (1952).
[4] J. Grey, “Multidimensional perceptual scaling of musical timbres,” J. Acoust. Soc. Am., vol. 61, no. 5, pp. 1270–
1277 (1977).
[5] D. L. Wessel, “Timbre space as a musical control
structure,” Comput. Music J., vol. 3, no. 2, pp. 45–52 (1979).
[6] S. McAdams, W. Winsberg, S. Donnadieu, G. De
Soete, and J. Krimphoff, “Perceptual scaling of synthesizedmusical timbres: Common dimensions, specificities, andlatent subject classes,” Psychol. Res., vol. 58, pp. 177–192(1995).
[7] S. Lakatos, “A common perceptual space for harmonic and percussive timbres,” Percept. Psychophys.,vol. 62, no. 7, pp. 1426–1439 (2000).
[8] J. W. Beauchamp, “Perceptually correlated parameters of musical instrument tones,” Arch.Acoust., vol. 36, no.
2, pp. 225–238 (2012).
[9] J. Blauert and U. Jekosch, “A layer model of soundquality,” J. Audio Eng. Soc., vol. 60, no. 1/2, pp. 4–12
(2012).
[10] A. B. Horner, J. W. Beauchamp, and R. H. Y. So,“A search for best error metrics to predict discrimination of original and spectrally altered musical instrument
sounds,” J. Audio Eng. Soc., vol. 54, pp. 140–156 (2006Mar.).
[11] A. B. Horner, J. W. Beauchamp, and R. H. Y. So,“Detection of time-varying harmonic amplitude alterations
due to spectral interpolations between musical instrument
tones,” J. Acoust. Soc. Am., vol. 125, no. 1, pp. 492–502(2009).
[12] M. Hall and J. Beauchamp, “Clarifying spectral
and temporal dimensions of musical instrument timbre,”Acoust. Can. J. Can. Acoust. Assoc., vol. 37, no. 1,
pp. 3–22 (2009).
[13] S. Barrass, “A perceptual framework for the auditory display of scientific data,” ACM Trans. Appl. Percept.,
vol. 2, no. 4, pp. 389–402 (2005).
[14] R. Plomp, Aspects of Tone Sensation: A Psychophysical Study, ch. 6 (Timbre of Complex Tones),
pp. 85–110 (Academic Press, New York, 1976).
[15] W. Slawson, Sound Color, pp. 3–21 (University ofCalifornia Press, Berkeley, CA, 1985).
683

TERASAWA ET AL.

[16] H. F. Pollard and E. V. Jansson, “A tristimulus
method for the specification of musical timbre,” Acustica,
vol. 51, pp. 162–171 (1982).
[17] J. S. Bridle and M. D. Brown, “An experimental
automatic word-recognition system: Interim report,” JSRU
Report 1003, Joint Speech Research Unit, 1974.
[18] P. Mermelstein, “Distance measures for speech
recognition, psychological and instrumental,” in Pattern
Recognition and Artificial Intelligence (C. H. Chen, ed.),pp. 374–388 (Academic Press, New York, 1976).
[19] S. B. Davis and P. Mermelstein, “Comparison ofparametric representations for monosyllabic word recognition in continuously spoken sentences,” IEEE Trans. Speech
Audio Process., vol. ASSP-28, pp. 357–366 (1980 Aug.).
[20] L. Rabiner and B.-H. Juang, Fundamentals of
Speech Recognition, pp. 183–190 (Prentice Hall, Upper
Saddle River, NJ, 1993).
[21] G. D. Poli and P. Prandoni, “Sonological models
for timbre characterization.,” J. New Music Res., vol. 26,
pp. 170–197 (1997).
[22] J.-J. Aucouturier, Ten Experiments on the Modellingof Polyphonic Timbre. Ph.D. thesis (University of Paris 6,
Paris, France, 2006).
[23] S. Heise, M. Hlatky, and J. Loviscach, “Aurally andvisually enhanced audio search with soundtorch,” in ACM
CHI 2009 Extended Abstracts, pp. 3241–3246 (2009 Apr.).
[24] N. Osaka, Y. Saito, S. Ishitsuka, and Y. Yoshioka,“An electronic timbre dictionary and 3d timbre display,”in Proc. 2009 Int. Computer Music Conference, pp. 9–12
(2009).
[25] M. Hoffman and P. R. Cook, “Feature-based synthesis for sonification and psychoacoustic research,” in Proc.
12th Int. Conf. Auditory Display, London, UK., pp. 254–257(2006).
[26] A. B. Horner, J. W. Beauchamp, and R. H. Y. So,“Evaluation of mel-band and mfcc-based error metrics forcorrespondence to discrimination of spectrally altered musical instrument sounds,” J. Audio Eng. Soc., vol. 59, no. 5,
pp. 290–303 (2011).
[27] H. Terasawa, A Hybrid Model for Timbre Perception: Quantitative Representations of Sound Color andDensity. Ph.D. thesis (Stanford University, Stanford, CA,
Stanford, CA, 2009).
[28] H. Terasawa, M. Slaney, and J. Berger, “Perceptual
distance in timbre space,” in Proc. ICAD 05 - Eleventh
Meeting of the International Conference on Auditory
Display, pp. 61–68 (2005).
[29] H. Terasawa, M. Slaney, and J. Berger, “A timbre
space for speech,” in Proc. Interspeech 2005–Eurospeech,
pp. 1729–1732, 2005.
[30] H. Terasawa, M. Slaney, and J. Berger, “The thirteen colors of timbre,” in Proc. IEEE Workshop on Ap-

684

PAPERS

plications of Signal Processing to Audio and Acoustics,
pp. 323–326 (2005).
[31] E. Zwicker and H. Fastl, Psychoacoustics – Facts
and Models, pp. 223–226 (Springer, Berlin, 1999).
[32] S. Shamma, “Speech processing in the auditory system,” J. Acoust. Soc. Am., vol. 78, no. 5, pp. 1612–1632,
1985.
[33] T. Irino and R. D. Patterson, “Segregating information about the size and shape of the vocal tract using
a time-domain auditory model: The Stabilised WaveletMellin Transform,” Speech Commun., vol. 36, pp. 181–203,
2002.
[34] A. Bregman, Auditory Scene Analysis, 2nd ed (MITPress, Cambridge, MA, 2001).
[35] S. Barrass and G. Walker, “Using sonification,”
Multimedia Syst., vol. 7, pp. 23–31 (1999).
[36] T. Hermann, G. Baier, U. Stephani, andH. Ritter, “Vocal sonification of pathologic EEG features,”in Proc. Int. Conf. Auditory Display (ICAD 2006), pp. 158–
163 (2006).
[37] R. Cassidy, J. Berger, K. Lee, M. Maggioni, and
R. R. Coifman, “Auditory display of hyperspectral colon
tissue images using vocal synthesis models,” in Proc. Int.
Conf. Auditory Display (ICAD 2004), pp. 1–8 (2004).
[38] J. Sundberg, “Vibrato and vowel identification,”Arch. Acoust., vol. 2, pp. 257–266 (1977).
[39] S. McAdams and X. Rodet, “The role of FMinduced AM in dynamic spectral profile analysis,” in Basic
Issues in Hearing (H. Duifhuis, J. Horst, and H. Wit, eds.),
pp. 359–369 (Academic Press, London; San Diego, CA,
1988).
[40] M. Slaney, “Auditory toolbox version 2,” Tech. Rep.
1998-010, Interval Research, 1998.
[41] W. Mendenhall and T. Sincich, Statistics for Engineering and the Sciences, pp. 531–698 (Prentice Hall,
Upper Saddle River, NJ, 1995).
[42] J. Rogers, K. Howard, and J. Vessey, “Using significance tests to evaluate equivalence between two experimental groups,” Psychological Bulletin, vol. 113, no. 3,
pp. 553–565 (1993).
[43] J. W. Beauchamp, H. Terasawa, and A. B. Horner,“Predicting perceptual differences between musical sounds:
A comparison of Mel-band and MFCC based metric results
to previous harmonic-based results,” in Proc. Soc. Music
Perception and Cognition 2009 Biennial Conference, p. 82(2009).
[44] V. Alluri and P. Toiviainen, “Exploring perceptualand acoustical correlates of polyphonic timbre,” Music
Percept., vol. 27, no. 3, pp. 223–241 (2009).
[45] E. Schubert and J. Wolfe, “Does timbral brightness
scale with frequency and spectral centroid?,” Acta Acust.
United Acust., vol. 92, pp. 820–825 (2006).

J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

PAPERSIN SEARCH OF A PERCEPTUAL METRIC FOR TIMBRE

THE AUTHORS

Hiroko Terasawa

Jonathan Berger

Hiroko Terasawa received B.E. and M.E. degrees inElectrical Engineering from the University of ElectroCommunications, Japan, and M.A. and Ph.D. degrees
in Music from Center for Computer Research in Musicand Acoustics (CCRMA), Stanford University, the UnitedStates. She is the recipient of the Centennial TA Award
from Stanford University (2006), the Artist in Residence
at Cité Internationale des Arts (2007), the second place of
the Best Student Paper Award in Musical Acoustics at the156th ASA Meeting (2008), the John M. Eargle Memorial Award from AES Educational Foundation (2008), theSuper Creator Award from ITPA Mitoh Program (2009),
and the JST-PRESTO Research Grant (2011). Her research interests include timbre perception modeling andtimbre-based data sonification. She is now a researcher atUniversity of Tsukuba and JST PRESTO, and a lecturer onelectronic music at Tokyo University of the Arts.

r

Jonathan Berger, The Denning Provostial Professor inMusic at CCRMA, Stanford University, is a composer andresearcher. He has composed orchestral music as well aschamber, vocal, and electro-acoustic and intermedia works.
Berger was the 2010 Composer in Residence at the Spoleto USA Festival, which commissioned a chamber work
for soprano Dawn Upshaw and piano quintet. He is currently working on a chamber opera commissioned by the
Andrew Mellon Foundation. Other major commissions andfellowships include the National Endowment for the Arts(a work for string quartet, voice, and computer in 1984,
soloist collaborations for piano, 1994, and for cello, 1996,
and a composers fellowship for a piano concerto in 1997);
The Rockefeller Foundation (work for computer-tracked
dancer, live electronics, and chamber ensemble); and The
Morse and Mellon Foundations (symphonic and chambermusic). Berger received prizes and commissions from theBourges Festival, WDR, the Banff Centre for the Arts,
Chamber Music America, Chamber Music Denver, theHudson Valley Chamber Circle, The Connecticut Commission on the Arts, The Jerusalem Foundation, and others.
Bergers recording of chamber music for strings, Miracles
and Mud, was released by Naxos on their American Masters series in 2008. His violin concerto, Jiyeh, is soon to
be released by Harmonia Mundis Eloquentia label. Bergers
research in music perception and cognition focuses on theformulation and processing of musical expectations, andthe use of music and sound to represent complex information for diagnostic and analytical purposes. He has authoredand co-authored over seventy publications in music theory, computer music, sonification, audio signal processing,

J. Audio Eng. Soc., Vol. 60, No. 9, 2012 September

Shoji Makino

and music cognition. Before joining the faculty at Stanford he taught at Yale where he was the founding directorof Yale University’s Center for Studies in Music Technology. Berger was the founding co-director of the Stanford
Institute for Creativity and the Arts (SICA) and, codirected
the Universitys Arts Initiative.

r

Shoji Makino received B.E., M.E., and Ph.D. degrees
from Tohoku University, Japan, in 1979, 1981, and 1993,
respectively. He joined NTT in 1981. He is now a Professor at University of Tsukuba. His research interests include
adaptive filtering technologies, the realization of acoustic echo cancellation, blind source separation of convolutive mixtures of speech, and acoustic signal processing
for speech and audio applications. He received the ICA
Unsupervised Learning Pioneer Award in 2006, the IEEE
MLSP Competition Award in 2007, the TELECOM System
Technology Award in 2004, the Achievement Award of theInstitute of Electronics, Information, and Communication
Engineers (IEICE) in 1997, and the Outstanding Technological Development Award of the Acoustical Society ofJapan (ASJ) in 1995, the Paper Award of the IEICE in 2005
and 2002, the Paper Award of the ASJ in 2005 and 2002.
He is the author or co-author of more than 200 articles
in journals and conference proceedings and is responsible
for more than 150 patents. He was a Keynote Speaker at
ICA2007, a Tutorial speaker at ICASSP2007, and a Tutorial speaker at INTERSPEECH2011. He has served onIEEE SPS Awards Board (2006–2008) and IEEE SPS Conference Board (2002–2004). He is a member of the James
L. Flanagan Speech and Audio Processing Award Committee. He was an Associate Editor of the IEEE Transactions
on Speech and Audio Processing (2002–2005) and is an
Associate Editor of the EURASIP Journal on Advancesin Signal Processing. He is a member of SPS Audio andElectroacoustics Technical Committee and the Chair of theBlind Signal Processing Technical Committee of the IEEE
Circuits and Systems Society. He was the Vice Presidentof the Engineering Sciences Society of the IEICE (2007–2008), and the Chair of the Engineering Acoustics Technical Committee of the IEICE (2006–2008). He is a memberof the International IWAENC Standing committee and amember of the International ICA Steering Committee. He
was the General Chair of WASPAA2007, the General Chairof IWAENC2003, the Organizing Chair of ICA2003, and is
the designated Plenary Chair of ICASSP2012. Dr. Makino
is an IEEE SPS Distinguished Lecturer (2009–2010), an
IEEE Fellow, an IEICE Fellow, a council member of theASJ, and a member of EURASIP.

685