Selected Papers

Timbre Synthesis Technology that Serves
Communication
Naotoshi Osaka, Ken-Ichi Sakakibara†,
and Takafumi Hikichi
Abstract
Sounds other than spoken language play an important role in communication. Moreover, there aredemands for high-quality timbre synthesis technology for creating multimedia contents. NTT Communication Science Laboratories is exploring technologies for synthesizing both musical instrument sounds
and singing voices. This paper introduces three technologies: i) a physical model of the “sho,” a Japanese traditional music instrument, ii) a model for synthesizing throat singing, and iii) a sound synthesis
system called “Otkinshi” as a tool for content creation.

1. Introduction
Sound information is important in human-tohuman communication. Awareness of this importance has increased recently, and, along with visual
information, sound is being recognized as vital to thecreation of multimedia content. This paper focuses onsound information other than speech as spoken language.
Sound information processing is still unfamiliar tomost people. However, sound and music information
processing will become popular, and even if you are
not well versed in music, you will soon be able tohandle and manipulate sound as you like. On theother hand, experts look forward to the developmentof sound representation technologies. Other possibilities include applications to both artistic music andcommercial contents such as games and multimedia
contents. The key to these applications is new timbre
synthesis technology. NTT Communication ScienceLaboratories are conducting various research projects
on implementing timbre synthesis technologies.
Two areas we are investigating at present are soundmorphing [1] and sound hybridization. In sound morphing, a sound is changed smoothly and continuous† NTT Communication Science Laboratories
Atsugi-shi, 243-0198 JapanE-mail: kis@brl.ntt.co.jp

Vol. 1 No. 2 May 2003

ly from one sound to another or an intermediate timbre between the timbres of two given sounds is synthesized. In sound hybridization, elements of sound(loudness, pitch, and timbre) are taken from different
sounds and a composite sound is synthesized.
We also are developing technology for synthesizing
known musical instrument sounds. By investigating
the sounding mechanism of musical instruments scientifically, we hope to understand a family of instruments or one instrument in particular. The advantageof this approach is that it lets us simulate musical performances. However, the technology goes beyond
simple synthesis of the original instrument sound: itwill enable us to synthesize impossible musical performances with, for example, infinitely fast finger
movement.
We are also researching singing voices. From theviewpoint of new timbre synthesis and control, ourgoal is timbre synthesis of singing voices based on
scientific investigation of the voicing mechanism insinging. These research topics are closely related toresearch on voicing mechanisms in spoken language
and general musical instrument sound synthesis, andthey should bridge the gap between music and speechresearch.
All of these technologies have been incorporated
into a sound synthesis system.

45

Selected Papers

2. Modeling the sound production mechanism ofthe sho
We think it important to research Japanese traditional musical instruments and model their soundproduction mechanisms because little scientificresearch has been done on such instruments. Furthermore, there is a need to understand why Japanesemusical instruments have not been modified toward
large volume, stable pitch, and quick control.
Besides, these instruments are not treated in computer music composition.
The three main instruments in Gagaku music are
the sho, hichiriki, and ryuteki [2]. From the viewpoint
Interaction
Interaction
Flow affects reed motion
Reed motion affects flow
Blowing pressure
control by a player

SourceReed
model

Open/close
finger-holes

Pipe resonancemodel

Resonator

Fig. 1. Physical model of the sho.

of musical acoustics, these resemble the harmonica,
oboe, and flute, respectively. The sho resembles theharmonica in that metal reeds, called free reeds, arethe source of sound generation, and it is played byblowing and drawing. However, it also has structuraldifferences and, unlike the harmonica, involves fingering. It is difficult to find a Western instrument thathas a similar mechanism. The sho was imported into
Japan more than a thousand years ago. In spite ofsuch a long history, no detailed research has been
done on its sound production mechanism.
We measured the relationships among reed vibration, resonance frequencies of the pipe, and soundpitch or timbre. Based on the results, we constructeda physical model of the sho [3] and synthesizedsounds by computer (Fig. 1). We have built a system
controlled by blowing pressure and fingering that has
a graphical user interface (Fig. 2). It can be used by
beginners for practice. When you input chord names,
their fingering positions are highlighted and you hear
their synthesized sounds. Thus, you can understandthe chords both visually and aurally.
Furthermore, by modifying the pipe lengths or reedcharacteristics, which is impossible with real instruments, the system can synthesize sounds with vibratoor tremolo. You can control parameters freely andseek your favorite sound timbres with dynamic features. Implementing the sound production mechanism on a computer in this way lets us extend the possibilities of conventional instruments and musicalcomposition.
3. Synthesis and control of timbre of the singing
voiceFig. 2. Control panel of our system.

46

Each style of singing in the world (traditional European singing, Japanese Minyoh, Korean Pansori,Tyvan Khoomei, and so on) is characterized by a timbre that is, for the most part, unique to it. These various timbres may share an important and extensive
domain in timbre space. On the other hand, the widerange of timbres in singing voices is produced bycontinuous control of the phonatory organs, which
are common to all human beings. Therefore, modeling their timbre production mechanisms by extracting
the parameters that govern the control of phonatory
movements based on physiological observation could
lead to a system for synthesizing and controlling thevarious timbres of singing voices.
The timbre of the human voice is varied by controlling both the voice source (i.e., airflow produced by
the vocal fold (VF) vibration) and the resonance cavNTT Technical Review

Selected Papers

ity, whose shape is changed by the motion of thearticulatory organs, such as the tongue, lips, andvelum (Fig. 3). The voice source controlled by laryngeal adjustment especially contributes to the varied
timbres of singing voices.
To model voice sources for timbre synthesis, we are
studying throat singing in collaboration with the University of Tokyo. Throat singing, typified by Tyvan
Khöömei and Mongolian Khöömij, is very different
from traditional European singing in timbre and voice
production mechanisms.
In throat singing, there are two different laryngeal
voices: the drone and kargyraa. The drone is sometimes called “damigoe” in Japanese and its perceptual impression can be classified as pressed. Moreover,
it is slightly different from the normal pressed voice.
The kargyraa is a very low pitched voice that ranges
out of the modal register.
We observed the laryngeal adjustments and movements of these two different laryngeal voices bysimultaneously recording sound waveforms, electroglottography (EGG) waveforms, and high-speed
digital images (4500 frames/s) through a flexible
endoscope inserted into the nose cavity of a singer[4], [5]. As a result, we clarified that the false vocal
folds (FVFs), which do not vibrate in normal phonation, vibrate and contribute to the generation of thespecial voice sources in throat singing (Fig. 4).
Based on these results and the laryngeal airflow

estimated by inverse filtering, we propose a newlaryngeal flow model as a signal model which is formulated by mathematical equations. By controllingits parameters, we can use this model to synthesize
various laryngeal sources, such as normal andpressed voices and the drone and kargyraa voices inthroat singing [6].
We also propose a new physical model of the laryngeal source called the 2 × 2-mass model, which is aself-oscillating model of the VF and FVF vibrations.
The model (Fig. 5) was devised by attaching a twomass model for FVFs to the ordinary two-mass model
Vocal tract
False vocal
fold

False glottis

Laryngeal
ventricle

Vocal fold
GlottisTracheaFig. 4. Human larynx (coronal section).
Radiated spectrum
Level

Vocal tract

Ug
Ag’2Ag’1

Frequency

r’2d’2

Velum

Vocal tract transfer function
Vocal tractr’1

Formantk’2

False vocal foldk’c
d’1

Level

k’1 m’1r2

Frequency

Level

Laryngeal voice source
spectrumd2
LarynxTrachea

r1k2

Vocal foldkc
d1

k1 m 1Ag2Frequency

Airflow

Laryngeal voice waveform

Lungs

Ps

Ag1

Trachea
TimeFig. 5. 2 × 2-mass model.
Fig. 3. Human speech organs and mechanism.

Vol. 1 No. 2 May 2003

47

Selected Papers

for VFs with a laryngeal ventricle space between themodels. By changing its control parameters, such as
mass and stiffness, we can use the 2 × 2-mass model
to synthesize various laryngeal voice sources [7].
4. Sound synthesis system Otkinshi

also displayed serially using operation icons. As aresult, sounds can be modified and resynthesizedusing icons. Moreover, users can re-run the recorded
and modified text for the first time. This function isvery convenient and efficient when, after creating aparticular sound, you want to have multiple sounds
repeatedly with slightly changed parameters. We aremaking preparations so that those who want to use theprototype system can download it from the Internet.
We are also preparing to provide the system toCCRMA (Center for Computer Research in MusicSound database

The name Otkinshi derives from the Japanese words
meaning sound and voice system. There are two versions: a Linux version for researchers and a Windows
version for ordinary users. Figure 6 shows the system
configuration. Techniques for soundmorphing, vibrato control (in whichGUI
vibrato is added to or subtracted froma musical sound), and physicalmodel-based sound synthesis areSound Synthesis•Special functions
implemented in software. General- Vibrato control
purpose functions include a wave edi- Morphing: physical model/signal model- Granular synthesis
tor and spectral editor, whose functions are equivalent to those of a com•Sound editor functions- Sound wave generation, recording, playback
mercial wave editor.
- Sound wave modification
Another feature of the system is its•Spectral editor functions- Various spectral display
user interface [8]. Previous systems- Decomposition of sound into the collection of partials
have had a clear distinction between- Selection, deletion of arbitrary partials
music software and timbre software.
•Performance
Therefore, music creators have had- No-delay sound playbackwith button press
to manipulate two types of software- Live performance use
separately. This inconvenience was- MIDI control
cleverly solved by defining a soundWindows PC
object hierarchically from the topFig. 6. System configuration.
(the music level) to the bottom (thetimbre level). Figure 7 shows thedata structure and the display of theLayer 0
Layer 1“sound object.” In the sound object, aLayer 0
music piece is represented by one
Layer 1
icon. Double clicking an icon brings
up a detailed panel. Each stage of thehierarchy, has an icon display (layer0) and a panel display of the details(layer 1). In the detailed display
panel, sound is expressed as a combiMultiple-soundobject
nation of multiple parts (channels).
Each part can be traced to the lower
layer by double clicking. The lowest
layer represents a sound wave disSingle-soundobjectplay. This straightforward expressionof music to sound was not available
before, and it makes the manipulation of both sound and music easier.
Layer 0: display with icon
: Multiple-sound object
The system stores a user’s operaLayer 1: display in detail: Single-sound object
tion history using the programming
language “Ruby.” The history isFig. 7. Hierarchical representation of sound objects.
48

NTT Technical Review

Selected Papersand Acoustics), Stanford University for reserch,music composition, and education.
5. Conclusions
In NTT Communication Science Laboratories, therange of research relevant to music timbre extends
from the science of sound to prototype system development. We expect to clarify various sound mechanisms and establish sound synthesis technology andsystems by applying. The results of our research willfind uses in content, such as games and broadcast programs, and artistic representation such as concert
music.
References
[1]
[2]

[3]

[4]

[5]

[6]

[7]

[8]

N. Osaka, “Timbre interpolation of sounds using a sinusoidal model,”ICMC 95 Proceedings, Banff, Canada, pp. 408-411, Sep. 1995.
S. Sadie ed., The New Grove Dictionary of Music and Musicians, 9,Japan, III, 1, pp. 510-515, Macmillan Publishers Limited, London,1980.
T. Hikichi, N. Osaka, and F. Itakura, “Time-domain simulation ofsound production of the sho,” Journal of Acoustical Society of America, Vol. 113, No. 2, pp. 1092-1101, 2003.
K.-I. Sakakibara, S. Adachi, T. Konishi, K. Kondo, E. Z. Murano, M.
Kumada, M. Todoroki, H. Imagawa, and S. Niimi, “Analysis of vocal
fold vibrations in throat singing,” Tech. Rep. Musical Acoust., Vol.
19, No. 4, pp. 41-48, Acoust. Soc. Jpn., Sep. 2002 (in Japanese).
http://www.acoustics.org/press/144th/Sakakibara.htm
K.-I. Sakakibara, T. Konishi, K. Kondo, E. Z. Murano, M. Kumada,H. Imagawa, and S. Niimi, “Vocal fold and false vocal fold vibrations
and synthesis of khoomei,” Proc. of ICMC, pp. 135-138, Sep. 2001.
K.-I. Sakakibara, T. Konishi, K. Kondo, E. Z. Murano, M. Kumada,H. Imagawa, and S. Niimi, “Vocal fold and false vocal fold vibrations
and synthesis of khoomei,” Proc. of ICMC, pp. 135-138, Sep. 2001.
K.-I. Sakakibara, H. Imagawa, S. Niimi, and N. Osaka, “Synthesis ofthe laryngeal source of throat singing using a 2 × 2-mass model,”Proc. of ICMC, pp. 5-8, Sep. 2002.
N. Osaka and T. Hikichi, “Visual manipulation environment forsound synthesis, modification and performance,” Proceedings of the1999 International Computer Music Conference, pp. 429-432, 1999.

Vol. 1 No. 2 May 2003

Naotoshi Osaka
Former Senior Research Scientist, leading
Media Representation Research Group, NTT
Communication Science Laboratories.
He received the M.S. degree in electrical engineering from Waseda University in 1978. His
main research interests include telephone transmission performance and speech dialogue. He
received a D.Eng degree for an objective modelfor telephone transmission performance. He iscurrently studying timbre synthesis for bothsounds and speech. He is currently a professor atTokyo Denki University. He is a senior member ofIEEE and is also a member of the IEICE, IPSJ andthe ASJ. He is also a vice-president of International Computer Music Association (ICMA).

Ken-Ichi Sakakibara
Research Scientist, Media Representation
Research Group, NTT Communication ScienceLaboratories.
He received a B.Sc. and M.Sc. in mathematics
from Kyoto University. He is currently interested
in the timbre of musical sounds, singing voices,and voice quality. He is a member of the Acoust.
Soc. Jpn., the Acoust. Soc. Am., the IPSJ, and theJpn Soc. of Logop. and Phoniat. and also a member of the board of directors of the Jpn. Association of Vocalization Instructors.

Takafumi Hikichi
Research Scientist, Media Information Laboratory, NTT Communication Science Laboratories.
He received the B.S. and M.S. degrees in electrical engineering from Nagoya University, in1993 and 1995, respectively. In 1995, he joined
NTT Basic Research Laboratories. He wasawarded the Awaya-Kiyoshi prize from theAcoustical Society of Japan in 2000. He is now
working at the NTT Communication ScienceLaboratories. Mr. Hikichi is a member of theAcoustical Society of America, the Institute ofElectronics, Information and Communication
Engineers, the Information Processing Society ofJapan, and the Acoustical Society of Japan.

49