Selected Papers

Human Speech Production Mechanisms
Masaaki Honda†
Abstract
We use language every day without devoting much thought to the process, but articulatory movement—the movement of the lips, tongue, and other organs—is among the subtlest and most adept of any
actions performed by human beings. Here, I discuss the mechanisms of speech production, introducefunctions for producing sound by controlling the movement of vocal organs, and consider models forachieving speech functions that will enable a computer to mimic a voice that it hears.

1. Introduction
Speech is a natural form of communication forhuman beings, and computers with the ability tounderstand speech and speak with a human voice are† NTT Communication Science Laboratories
Atsugi-shi, 243-0198 Japan
E-mail: gomi@idea.brl.ntt.co.jp

Human speech
production process

Motor control

expected to contribute to the development of morenatural man-machine interfaces. Computers with thiskind of ability are gradually becoming a reality,
through the evolution of speech synthesis and speech
recognition technologies. However, in order to givethem functions that are even closer to those of humanbeings, we must learn more about the mechanisms bywhich speech is produced and perceived, and develop speech information processing technologies that

Articulatory motion

Sound generation

Velum
Motor cortex

Lips

TongueJaw

GlottalsourceVocal-tract
resonance Speech

GlottisSpeech production
model
Phonemestrings

Control parameters

Motor planningmodel

Motor task

Trajectoryformation
model

Articulatory motion

Sound generation
model

Vocal-tract area

GlottisSpeechSpectrum

Lips

Speech waveform

Frequency
Production

PhonemeSpeech motor
task

Articulatory
parameters

Vocal-tract
parameters

Spectralparameters

Speech waveform
Perception

Fig. 1. Speech production process and the model.

24

NTT Technical Review

Selected Papers

make use of these functions. We use speech every day
almost unconsciously, but an understanding of themechanisms on which it is based will help to clarify
how the brain processes information and will also
lead to the development of more human-like speechdevices through the imitation of these functions bycomputers.
Figure 1 shows the process of human speech and acomputer model corresponding to it [1]. The mechanism of speech is composed of four processes: language processing, in which the content of an utterance is converted into phonemic symbols in thebrain’s language center; generation of motor commands to the vocal organs in the brain’s motor center;articulatory movement for the production of speechby the vocal organs based on these motor commands;
and the emission of air sent from the lungs in the formof speech. Here, I provide a brief explanation of thesemechanisms and introduce a model for achieving
these functions on a computer.
2. Mechanisms of speech production andspeech synthesis
First, in order to simplify this discussion, I will
explain how speech is created mechanically—that is,
the mechanisms of speech synthesis—and then discuss how speech synthesis is related to the mechanisms involved in producing human speech. The firstattempts to create speech using a machine date backto long before the birth of computers, to a speaking
machine created by Wolfgang von Kempelen in
1771. It consisted of a bellows and a sound tube thatsimulated the vocal tract (the shape of the mouth fromthe vocal cords to the lips and nose), along with thereed of a musical instrument installed at one end of
the sound tube. The flow of air sent from the bellows
created a sound source by causing the reed to vibrate,
and voices with a wide variety of tones were createdby changing the shape of the resonance tube, thus
modifying the resonance qualities of the sound tube.
This speaking machine, which simulated the mechanisms of speech production using this type of soundsource and vocal tract sound tube, became the model
for speech synthesis devices that followed. The
Vocoder (voice coder), developed by Homer Dudleyin the 1930s, took a different approach. Rather thanusing the shape of the vocal tract itself, it recreatedthe sound resonance characteristics of the soundwaves within the vocal tract by using electrical circuits as resonance filters. Many of the speech synthesis systems used widely throughout the world today
Vol. 1 No. 2 May 2003

utilize this principle through digital signal processing
on computers or LSIs [2]. This same vocoder principle is also utilized in the digital speech coding technologies used in modern mobile phones.
The vocoder comprises a sound source generator
and a voice tract resonance filter. When the soundsource is a voiced sound, a cyclic pulse string is produced; when the sound source is unvoiced, a noise signal is produced. The pitch of the voice is controlled by
the cycle of the cyclic pulse string. In the case of the
vocal tract resonance filter, differences in tone resulting from phonemes such as the Japanese syllables “a”
or “ka” are controlled by modifying resonance characteristics (voice spectrum characteristics) [2].
The principle of speech synthesis using a vocoder is
to simulate the process of human speech productionin the sense that the process of producing speech isexpressed by the production of a sound source andresonance of sound. In human speech production as
well, in the case of voiced sounds, the air flow sentfrom the lungs passes through an opening in the vocal
cords, whose vibration causes that air flow to be converted into cyclic puffs of air that then become sound.
In the case of unvoiced sounds like the syllable “sa,”the air flow then passes through a narrow space
formed by the tongue inside the mouth, a turbulent
flow of air is produced, and this is emitted as a noiselike sound. These sounds sources themselves do nothave any noticeable tone, but changing the position of
the mouth creates sound wave resonance that differs
depending on the shape of the vocal tract sound tube,
and this results in the creation of various tones. When
we speak, we move our jaws, tongue, and other partsof our mouth; in fact, this changes the shape of the
vocal tract by changing the position of the mouth, andthis in turn enables us to control sound resonancecharacteristics.
A speech production model that more directly simulates the physical process of human speech production comprises lungs, vocal cords, and the vocal tract.
The vocal cords are expressed as a simple vibration
model, and the pitch of the speech changes according
to adjustments in the tension of the vocal cords.
When the vocal cords close, their vibration results invoiced sounds; when they open, this vibration stops,
and unvoiced sounds result. The vocal tract model iscreated as a non-uniform sound tube with differingcross-sectional areas, and the transmission of soundwaves inside the sound tube is expressed by a digital
filter. Vocal tract resonance characteristics are controlled according to the cross-sectional area (vocal
tract area function) for various parts of the vocal tract
25

Selected Papers

up to the lips.

Palate shape

3. Mechanisms of articulatory movement
The speech production process has many levels,from the movement of vocal organs to the productionof sounds. There are four hierarchical levels in speech
production: the speech sound level, vocal tract shapelevel, vocal organ configuration level, and muscle
contraction level. There is a “one-to-many” relationship among levels starting from the sound level andmoving toward the muscle contraction level. For
example, as is evident in the case of ventriloquism,
sounds that seem very similar can be created using
different vocal tract shapes. Similar vocal tract shapes
can be created using differing vocal organ configurations; for example, the degree of mouth opening isdetermined by the relative position of both lips andthe jaw. Furthermore, each individual vocal organ
involves two or more competing muscles, and the
vocal organ configuration is determined by their relative degree of contraction. This means that whenspeech with a given tone is created, the “one-tomany” relationship that exists among levels cannot be
determined uniquely because there is always an
excessive degree of freedom on the lower level.
This raises the questions of “What are the goals ofarticulatory movement?” and “How do humans determine specific human articulatory movements in thecontext of speech systems where these excessive
degrees of freedom exist?” There is no doubt that thegoal of articulatory movement is to transmit mainly
language information through speech. It is also natural to assume that in speech systems where a “one-tomany” hierarchical relationship exists, the goal willbe the highest level. In other words, we can assume
that the goal in the context of articulatory movement
is not to assign the position of the individual vocal
organs (e.g., the jaw, lips, or tongue), but rather toform the vocal tract that is most suitable for emitting
a given sound and to achieve the acoustic phenomenon for speech. This type of approach can be seen
within actual human articulatory movement.
Figure 2 shows the action of the tongue when the
shape of the palate is changed using an artificial
palate in the case of an utterance of the Japanese syllable “sha” [2]. Even when the shape of the upper jaw
changes, the shape of the tongue responds instantaneously, changing to achieve the normal utterance ofthe syllable “sha.” The results of these experimentssuggest that the goal of articulatory movement is not
only the absolute position of the various individual
26

T2

T3

T4

T1
J
Modified palate shape

T4
T2

T3

T1
J

Fig. 2. Trajectories of jaw (J) and tongue (T1, T2, T3, T4)
movements in production of /sha/.

vocal organs, but also the shape of the vocal tract created by their relative positions.
Another feature of articulatory movement is that
this is not a simple movement, but continuous movement for the purpose of uttering continuous sounds.
For example, the movement of the mouth when uttering the sound “api” is not a simple connection of theindividual mouth positions (articulations) for thephonemes /a/, /p/, and /i/. The articulation of the consonant /p/ is affected by the articulation of the vowel/i/ that follows it, and when it is uttering /p/, thetongue is already in position for the vowel /i/. The
term “coarticulation” is used to refer to the phenomenon in which the mouth position for individualphonemes in the utterance of continuous sounds
incorporates the effects of the mouth position forphonemes uttered immediately before and after. Thisis a very significant feature of articulatory movement.
4. Speech production models
Now, let me discuss the mechanisms of speech production models. As shown in Fig. 3, in a speech production model, a phoneme symbol (equivalent to apronunciation key) for a spoken word is specified, amotor goal (task) for the articulatory movement isgenerated, and a motor trajectory for the vocal organis calculated for the motor task assigned. The vocal
tract shape is determined from the position of the
vocal organs, and speech is produced by controllingthe speech production model using the vocal tract
NTT Technical Review

Selected Papers

Phoneme string (/onsei/)

Speech motor task

Trajectory formation
X

Articulatoryvariable Space

Zo

Zn

Zs

t1

t2

t3

Time
0

T

Optimal Trajectory

Fig. 3. Trajectory formation model of articulatory movements.

area [4].
Motor tasks are expressed by vocal tract variables
that represent the local shape of the vocal tract. The
vocal tract variables are: opening of the lower jaw,
opening and protrusion of the lips, size and positionof the narrowed segment of the vocal tract as formedby the tongue, and opening of the velum palatinum,
which adjusts the connection to the nasal cavity in the
vocal tract. The method of specifying the motor taskdiffers depending on the phoneme. For example, inthe case of /p/, only the opening of the lips and thevelum palatinum are specified; other vocal tract variables are not. Vocal tract variables are expressed as afunction of articulation variables that represent theconfiguration of individual vocal organs. For example, the vocal tract variable for the opening of the lips
is a function of the jaw position and the height of both
lips, and it is determined by the relative position of
the jaw and lips. Each vocal tract variable is thus
functions of multiple articulation variables, and there
is a “one-to-many” relationship between them. For
this reason, articulation variables that represent theconfiguration of vocal organs cannot be uniformly
determined simply by specifying vocal tract variables. Furthermore, motor tasks are specified discretely in time as action targets, so there are an infinite number of motor trajectories for articulatory
movement that fulfills the motor tasks. Therefore, inorder to decide the articulatory movement, it is necVol. 1 No. 2 May 2003

essary to define a trajectory actually selected in thecontext of human articulatory movement from among
an infinite number of motor trajectories that satisfy
the motor task specified by the vocal tract variables.
As illustrated in Fig. 3, when the motor task is represented as partial spaces within an articulation variable
space, this issue of trajectory production is formulated as a problem of deciding the motor trajectory thatpasses through each partial space. In general, this isseen as a problem of indeterminacy, and the definition of a solution requires the introduction of new
constraints.
Here, the cost function is assumed to be the volume
defined as the weighted sum of [(the time derivativeof motive force on the speech dynamic system)2 +(motor speed)2], and the motor trajectory is decided
based on the constraint of minimizing this cost function. Qualitatively speaking, this motor index corresponds to the selection of the shortest possible trajectory while avoiding the use of unnecessary motive
force energy for articulatory movement.
5. Speech mimicking model
A human can imitate the sound of an utterance hehas heard, without necessarily being able to speakthat particular language. This means that the person
can estimate which mouth movements are necessary
to produce a sound similar to the one he has heard.
27Selected Papers

For a computer to achieve this type of “HearingMimicking Speech” function, it must estimate themouth movement from the relevant speech. Generally, however, there is a one-to-many relationshipbetween the acoustic quality of the speech and theshape of the sound tube corresponding to the vocal
tract, so the position of the mouth cannot be uniquelydetermined from the utterance. In order to resolve thisproblem of indeterminacy, we must take into consideration the fact that the geometric shape of the soundtube is subject to constraints in terms of the positions
that can be taken by the vocal organs and the fact thatthe changes in the sound tube over time are subject toconstraints related to the movement of the vocal
organs.
Figure 4 illustrates the process flow when estimating the movement of the vocal organs at the time ofan utterance from the relevant speech [5]. The frequency of the input speech is analyzed, and the timeline of the spectrum is derived. The articulatory
movement and sounds are simultaneously observedin advance using an electromagnetic sensor system,

and articulation/sound data—consisting of pairs
composed of movement patterns for speech fragments about 100 ms long and temporal patterns forcorresponding speech spectra—is stored in a database. The spectrum timeline for the input speech ismatched with spectrum segment patterns in the database using spectrum segments of about 100 ms as thereference unit, and several patterns are selected fromthe database in order of their inter-pattern distance,from smallest to largest. Next, a determination ismade regarding the movement pattern series with thehighest degree of temporal continuity from among
the selected movement pattern candidates, based on
the constraint of temporal continuity as seen in articulatory movement.
In this method, a constraint related to the position
and movement of the mouth that could actually takeplace is incorporated into a database created using
observed articulatory motor data and speech data. In
this way, it is possible to estimate the movement of
the mouth from the speech with an average accuracyof about 1.9 mm. We have also confirmed that it is(Frequency)

Input speech

Spectralsequence(Time)
Spectral segment matching

Articulatory-acoustic
pair database

Path search ofarticulatory movements

Estimated articulatory
movements

Time( : Candidate ofarticulatory parameters)

•••

Lips
Jaw
Tongue
Velum(Time)

Fig. 4. Estimation of articulatory movements from speech acoustics.

28

NTT Technical Review

Selected Papers

possible to recreate the speech based on the mouth
movement estimated from the original speech using aspeech production model and to synthesize speechwith tones that are extremely similar to the original
speech. The model discussed here is a very close simulation of human “hearing-mimicking speech” functions and indicates that the functions demonstrated byhuman beings can be achieved even on computers.
6. Conclusion

Masaaki Honda
He received the B. E. M. E. and D. Eng.
degrees from Waseda University, Japan, in 1973,
1975, and 1978, respectively. Since 1978, he has
been with NTT Basic Research Laboratories,
where he has been involved in research on digitalspeech coding and speech signal modeling from1978 to 1988, and on speech production mechanism and the computational model from 1989 to
2003. Currently, he is the former group leader of
Speech and Motor Control research group of
Human Information Science Laboratory. Since
1999, he has also been a research group leader of
CREST project of Japan Science and Technology, where he has been involved in research on
motor planning of speech production.

I have provided a brief explanation of the mechanisms of speech production from the perspectives ofspeech-sound generation and articulatory movement
and illustrated how these functions could be achieved
on a computer. Currently, we are trying to develop a
speaking robot by constructing a biomechanicalvocal organ model that further incorporates the biological structures of human vocal organs and, at thesame time, by clarifying the motor control mechanisms for speech that are executed by the brain [6].
References
[1]

[2]
[3]

[4]

[5]

[6]

M. Honda, NTT CS Laboratories, Speech synthesis technology basedon speech production mechanism, How to observe and mimic speech
production by human, Journal of the Acoustical Society of Japan, Vol.
55, No. 11, pp. 777-782, 1999 (in Japanese).
S. Saito and K. Nakata, Fundamentals of Speech Signal Processing,Ohm Publishing, 1981 (in Japanese).
M. Honda, H. Gomi, T. Ito and A. Fujino, NTT CS Laboratories,
Mechanism of articulatory cooperated movements in speech production, Proceedings of Autumn Meeting of the Acoustical Society ofJapan, Vol. 1, pp. 283-286, 2001 (in Japanese).
T. Kaburagi and M. Honda, NTT CS Laboratories “A model of articulator trajectory formation based on the motor tasks of vocal-tract
shapes,” J. Acoust. Soc. Am. Vol. 99, pp. 3154-3170, 1996.
S. Suzuki, T. Okadome and M. Honda, NTT CS Laboratories, “Determination of articulatory positions from speech acoustics by applying
dynamic articulatory constraints,” Proc. ICSLP98, pp. 2251-2254,
1998.
M. Honda, NTT CS Laboratories, “Task planning mechanism ofspeech motor control, Japan Science and Technology Corporation,
CREST Project,” Journal of the Acoustical Society of Japan, Vol. 56,
No. 11, pp. 771, 2000 (in Japanese).

Vol. 1 No. 2 May 2003

29