A REAL-TIME FILLED PAUSE DETECTION SYSTEMFOR SPONTANEOUS SPEECH RECOGNITION
Masataka Goto

Katunobu Itou

Satoru Hayamizu

Machine Understanding Division, Electrotechnical Laboratory
1-1-4 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN.
{goto, kito, hayamizu}@etl.go.jp
http://www.etl.go.jp/˜goto/

ABSTRACTThis paper describes a method for automatically detecting filled(vocalized) pauses, which are one of the hesitation phenomenathat current speech recognizers typically cannot handle. The detection of these pauses is important in spontaneous speech dialogue systems because they play valuable roles, such as helping a speaker keep a conversational turn, in oral communication. Although a few speech recognition systems have processed
filled pauses within subword-based connected word recognition
or word-spotting frameworks, they did not detect the pauses individually and consequently could not consider their roles. In thispaper we propose a method that detects filled pauses and word
lengthening on the basis of small fundamental frequency transition and small spectral envelope deformation under the assumption that speakers do not change articulator parameters during
filled pauses. Experimental results for a Japanese spoken dialogue corpus show that our real-time filled-pause-detection system yielded a recall rate of 84.9% and a precision rate of 91.5%.
Keywords: Filled pause, Hesitation, Spontaneous speech

1

INTRODUCTION

The goal of this research is to improve the computer’s ability to understand speech to a degree that will make possible natural multimodal communication between humans
and computers. This requires the computer to recognize
the audio signals comprising the spontaneous speech uttered by a speaker thinking about speech contents on the
fly. Hesitation phenomena, such as filled (vocalized) or
unfilled (silent) pauses, word lengthening, restarts, andfalse starts, occur frequently in such speech. As an initial step toward dealing with those natural and inevitablephenomena, in this paper we concentrate on two frequent
phenomena, filled pauses and word lengthening, because
these phenomena play the same valuable roles in oral communication, such as helping a speaker hold a conversational turn and express mental and thinking states. In order to improve speech dialogue systems, we think that it isimportant to make good use of such roles without simply
neglecting those phenomena.
Typical HMM-based speech recognizers accept only
fluent read or planned speech without hesitation phenomena and have difficulty in dealing with spontaneous
speech. The phone model, for example, does not work
well when applied to speech with filled pauses and word
lengthening because the duration of a phone tends tolengthen suddenly, and the language model is not effective
enough to deal with filled pauses because the pauses can beinserted at almost arbitrary word positions. A few previous speech recognition systems [1][2][3] have partly handled filled pauses within subword-based connected word
recognition or word-spotting frameworks. One HMMbased recognizer [2], for example, added several frequentfiller words to the system vocabulary and another one [3]regarded filler words as out-of-vocabulary words and dealt
with them by using a subword-unit based decoder for processing unknown words. Those systems, however, did notdetect filled pauses individually and could not consider theroles of these pauses.
We therefore believe that it is necessary to detect
filled pauses (fillers) and word lengthening in spontaneous
speech by using bottom-up acoustical analysis. Previous investigations [4][5] of the prosodic features of filled
pauses suggested the feasibility of detecting the pauses.
The report of Quimbo et al. [5], in particular, supported
the bottom-up approach of analyzing prosodic features bypointing out that human beings can, from prosodic cues,
recognize filled pauses in speech that is in unfamiliar foreign language. In those investigations, however, a computational system of automatically detecting filled pauses
was not built.
In this paper we propose a method for detecting filled
pauses and word-lengthening phenomena in spontaneous
speech. Since both these hesitation phenomena have similar acoustical features and can be considered to have thesame functions in terms of oral communication, in the restof this paper we use the term “filled pause” for both. In
the following sections, we first discuss the roles of filled
pauses and describe the algorithm of the method. We thenshow experimental results obtained using our real-timesystem based on the proposed method. Finally, we discuss applicability of the method in a speech recognition
framework.

2 IMPORTANCE OF FILLED PAUSESIn this research, we hypothesize that the essential reason
that filled pauses are inevitable in spontaneous utterancesis that they are uttered when the thinking process cannot
keep up with the speaking process. When the speed of
speaking becomes faster than the speed of preparing its
content, a speaker uses filled or unfilled pauses until thenext speech content resulting from the thinking process arrives at the speaking process.

ESCA, Eurospeech99. Budapest, Hungary. ISSN 1018-4074, Page 227

Speech audio signals

The primary utility of detecting filled pauses is that it
makes it possible to improve the performance of speech
recognizers by avoiding the application of typical HMMbased recognition to the filled-pause periods in spontaneous speech. Furthermore, this detection enables speech
dialogue systems to make use of the following two important functions of filled pauses, which functions were also
discussed in [6][7].

Calculating instantaneous frequencies
Extracting frequency components
Estimating fundamental frequency
Estimating spectral envelope

Obtaining two featuresof filled pauses
F0 transition Spectral envelope deformation• Communicative functions
In spoken dialogue, a speaker uses filled pauses tokeep a conversational turn while taking enough thinking time to prepare a subsequent utterance. On theother hand, a listener hearing filled pauses usuallywaits for the speaker’s subsequent utterance without
interrupting the turn.
• Affective and cognitive functions
For achieving smooth dialogue by sharing mental
states among interlocutors, a speaker unconsciously
uses filled pauses to express mental states such as diffidence, anxiety, hesitation, and humility and also to
express different thinking states, such as retrieving
information from memory and seeking an expression
appropriate for a listener. On the other hand, a listener interprets filled pauses as indicators for inferring speaker’s mental and thinking states. In addition,
filled pauses sometimes enable a listener to predict
the speaker’s subsequent utterance to some extent.

3 DETECTION METHOD
The basic idea of our method is to find acoustical features of filled pauses in speech signals by using frequency
analysis. If filled pauses are, as described in the previous section, uttered while the speaking process is waiting
for the next speech content from the thinking process, a
speaker cannot change articulator parameters during thefilled pauses because subsequent utterances have not yet
been prepared. Our method hence assumes that a filled
pause contains a continuous voiced sound of an unvaried
phoneme, because such a sound is uttered when the vocal cords are vibrated with almost constant articulator parameters (i.e., with a constant vocal-tract shape). TypicalJapanese fillers such as /ee-/, /maa-/, and /ano-/ as well as
most word-lengthening sounds satisfy this assumption.
Our method accordingly detects filled pauses on the basis of the following two features:
1. Small F0 (fundamental frequency) transition
When the tension of the vocal cords is unvaried underconstant articulator parameters, the F0 of the voice
remains almost constant.
2. Small spectral envelope deformation
When the vocal tract shape is unvaried under constantarticulator parameters, the spectral envelope forming
the formants remains almost constant. When the deformation of the envelope is evaluated, it is necessary to eliminate the air flow’s amplitude modulation,
since the air flow from the lungs may vary.

Evaluating possibility of filled pauses
Possibility of filled pauses
Filled-pause period

Figure 1: Overview of our filled-pause-detection method.

In the following, we describe the main procedure of our
method (Figure 1).

3.1 Calculating Instantaneous Frequencies
The first step is to calculate the instantaneous frequency[8], the rate of change of the phase of a signal, of filterbank outputs by using the short-time Fourier transform(STFT) whose output can be interpreted as a collection of
uniform-filter outputs. When the STFT of a signal x(t)with a window function h(t) is defined as ∞X(ω, t) =x(τ )h(τ − t)e−jωτ dτ = a + jb, (1)−∞the instantaneous frequency λ(ω, t) is given by this equation [8]:a ∂b − b ∂a(2)λ(ω, t) = ω + ∂t2 2∂t .
a +bIn our current implementation the input signal is digitizedat 16 bit / 16 kHz, and then the STFT with a 1024-sample
Hanning window is calculated by using the Fast Fourier
Transform (FFT). Since the FFT frame is shifted by 160
samples, the discrete time step (1 frame shift1 ) is 10 ms.

3.2 Extracting Frequency Components
The extraction of frequency components is based on thethe mapping from the center frequency ω of an STFT
filter to the instantaneous frequency λ(ω, t) of its output[9][10][11]. By finding fixed stable points of the mapping,
we can extract a set Ψf (t) of instantaneous frequencies ofthe frequency components by using the following equation[10]:
Ψf (t) = { ψ | λ(ψ, t) − ψ = 0,∂(λ(ψ, t) − ψ) < 0}.
(3)∂ψ
By calculating the power of those frequencies which is
given by the STFT power spectrum at Ψf (t), we can define
the power distribution function Ψp (ω, t) as| X(ω, t) | if ω ∈ Ψf (t)(4)Ψp (ω, t) =0otherwise.
1
The term time in this paper is the time measured in units offrame shift.

Page 2283.3 Estimating Fundamental Frequency

3.5 Obtaining Two Features of Filled Pauses

To estimate the F0 of a speaker’s voice in real-world audio signals containing background noises or music, we findthe most predominant harmonic structure in the extracted
frequency components by using a comb-filter-like analysis. The basic idea is to evaluate the possibility PF 0 (F, t)of the F0 at frequency F at time t. Hereafter, we use thefrequency unit cent to denote the log-scale frequency. In
this paper the frequency fHz in Hz is converted to the frequency fcent in cent as follows:

The two filled-pause features the method obtains are theamount Af (t) of the F0 transition that indicates how much
the F0 changes and the amount As (t) of the spectral envelope deformation that indicates how much and how unevenly the spectral envelope changes.
The amount Af (t) is defined as the absolute value ofthe slope bF 0 of a straight line obtained by least-squares
fitting of short-term transition of the log-scale F0 FF 0 (t).
The bF 0 is obtained by minimizing

fHz
fcent = 1200 log2 REF
HzREFHz = 440 × 2

3
12 −5

.

The F0 possibility PF 0 (F, t) is defined as ∞p(x; F ) Ψp (x, t) dx,
PF 0 (F, t) =

(5)
(6)

(7)

−∞

where the unit of frequencies x and F is cent, p(x; F )denotes a filter function, and Ψp (x, t) is the same as thepower distribution Ψp (ω, t) (Equation (4)) except that thefrequency unit is cent. The p(x; F ) passes harmonic components of the F0 F and is given byp(x; F ) =N

Hh−1 G(x; F + 1200 log2 h, W1 )

(8)

h=1


2
,
G(x; m, σ) = √ 1 2 exp − (x−m)
2σ 2
2πσ

(9)

where N (8, in our current implementation) is the numberof harmonics considered, H (0.97) is an amplitude attenuation factor, and W1 (20 cent) is the standard deviation of
the Gaussian distribution G(x; m, σ).
Finally, the frequency FF 0 (t) of the F0 is determined byfinding the frequency that maximizes PF 0 (F, t):
FF 0 (t) = argmax PF 0 (F, t).

(10)

F

F0 −1
errF 0 2 = ΣPeriod(FF 0 (t − τ ) − (aF 0 + bF 0 τ ))2 (12)τ =0over aF 0 and bF 0 , where PeriodF0 (5 frame shifts) is thefitting period.
The amount As (t) is defined as
 1 Nmax 1 Nmax
As (t) = NmaxΣn=1 bs (n)2 NmaxΣn=1 errs (n)2 , (13)where bs (n) is the slope of a straight line that is similarly obtained by least-squares fitting of n-th-harmonic’s
short-term transition of the log-scale power of the envelope Env(n,t). The method minimizes the fitting error
errs (n)errs (n)2 =(14)Periods −1(10 log10 Env(n, t − τ ) − (as (n) + bs (n)τ ))2Στ =0
over as (n) and bs (n), where Periods (10 frame shifts) is
the fitting period.

3.6 Evaluating Possibility of Filled Pauses
On the basis of short-term averages Si (t) (i = f, s) of
the two obtained features, the possibility Pf p (t) of filled
pauses is defined as
 (R Sf (t) + (1−R) Ss (t))2 (15)
Pf p (t) = exp −W2
Period −1

3.4 Estimating Spectral Envelope
For robustness in real-world environments, the spectral envelope is estimated by using only local information on theharmonic structure of the obtained F0 FF 0 (t). The power
P ow(k, t; FF 0 (t)) of k-th harmonic component of FF 0 (t)is extracted by finding the local-maximum power calculated with a Gaussian kernel around each F0’s multiple:
P ow(k, t; FF 0 (t)) =max G(x; FF 0 (t) + 1200 log2 k, W2 ) Ψp (x, t) (11)xwhere W2 (35 cent) is the standard deviation of the Gaussian distribution.
We then estimate the spectrum envelope in the linearscale frequency by linear interpolation of the adjacent
P ow(k, t; FF 0 (t)). This envelope is calculated under theupper-limit frequency (3200 Hz) that covers the first andsecond formant frequencies of Japanese vowels. To makeuse of the global envelope deformation as the filled-pausefeature, the method resamples the interpolated envelope at
low frequency resolution ξ (200 Hz) and obtains the spectral envelope Env(n, t) at frequency nξ where 1 ≤ n ≤Nmax (15). Then the Env(n, t) is normalized so that it satmax
isfies ΣN
n=1 Env(n, t) = 1 in order to compensate for theamplitude modulation of lungs’ air flow.

1Στ =0 fp
Si (t) = Periodfp

Ai (t − τ )(i = f, s)(16)

where R (0.034) and W (0.575) are empirical constant values and Periodfp (10 frame shifts) is the averaging period.
When this possibility is high enough for a certain periodof time, the method judges that a filled pause is uttered.
We calculate the accumulated sum Sumf p (t) of Pf p (t) as
long as Pf p (t) > e−1 . When Sumf p (t) > Thfp , where
Thfp (7e−1 ) is a constant threshold, the current time t isjudged to be within the filled-pause period.

4 EXPERIMENTAL RESULTS
A real-time filled-pause-detection system based on theabove method has been implemented and tested on a
Japanese spontaneous speech corpus consisting of 100
utterances by five men and five women (10 utterances
per subject). Each utterance contained at least one filled
pause. Those utterances were excerpted from a spontaneous speech dialogue corpus [12] collected using a Wizard of OZ system and were automatically segmented bydetecting each silence interval longer than 300 ms.
In our experiment the recall rate (the number of filled
pauses detected correctly / the total number of filled
pauses) was 84.9 percent (107 / 126) and the precision rate

Page 229

Figure 2: An example of the F0 and intermediate results (left)
and the corresponding spectral envelope (right) for part of amale spontaneous utterance /iqkaini-/.

Figure 3: An example of the hand-labeled phone sequence andthe detected filled-pause period for a male spontaneous utterance/iqkaini-#arimasune/.

Figure 4: An example of the original alignment result and thealignment result improved by the detected filled-pause period.

(the number of filled pauses detected correctly / the total
number of filled pauses detected) was 91.5 percent (107 /
117). Figure 2 shows an example of intermediate resultsof evaluating the possibility of filled pauses and Figure 3
shows an example of the result of correctly detecting afilled pause.
The main reasons for the recall-rate errors (misses) were
too short duration of filled pauses like a short /e-/, toolarge F0 changes, and disorder of harmonic components ofhoarse voices. The recall rate was exceptionally low (53.8
percent) for a particular male subject who tended to speakwith a low-frequency hoarse voice. On the other hand, theprecision-rate errors (false alarms) mainly occurred at continuous unvaried voiced sounds within words uttered witha flat F0. Such sounds tended to occur at successive similar vowel sounds caused by the target undershoot of phonetransition, while our method rejected typical successive
similar vowel sounds because their F0 changed enough inusual.

5 APPLICABILITY IN SPEECHRECOGNITION FRAMEWORKAs a preliminary step toward making use of the filledpause detection method in a speech recognition framework, we tested how much phone alignment can be improved by using the detected filled-pause periods. In
this preliminary test, the input was matched with a phone
HMM connected according to the given correct phone sequence corresponding to the input. In general, a filled
pause has a bad influence on the phone alignment determined by the Viterbi algorithm. To improve this align-ment, we used the detected filled-pause period for dynamic phone-duration control: during the detected period
we inhibited the transition from a vowel phone to the next
phone.
Figure 4 shows an example of an original bad alignment result and the alignment result improved by using
the filled-pause detection. Results like this suggest thatwhen utterances contain filled pauses the performance ofa typical HMM-based speech recognizer can be improvedby using the filled-pause detection method.

6 CONCLUSION
We have described a method for detecting filled pauses
and word-lengthening phenomena by finding a continuous voiced sound of an unvaried phoneme. The method isbased on two acoustical features, small F0 transition andsmall spectral envelope deformation, which are estimatedby identifying the most predominant harmonic structurein the input. Experimental results for a Japanese spontaneous speech corpus show that our system can detect, inreal time, filled pauses with a recall rate of 84.9 percent
and a precision rate of 91.5 percent.
We plan to apply our method to a speech recognizer
by using not only the filled-pause period (discrete judgement) but also the filled-pause possibility value (continuous judgement). Future work will also include applicationof our method to English filled pauses and integration of
the method with a speech dialogue system to make full useof the valuable functions of filled pauses.

REFERENCES[1] W. Ward. Understanding spontaneous speech: The Phoenix
system. In Proc. of ICASSP 91, pp. 365–367, 1991.
[2] S. Nakagawa and S. Kobayashi. Phenomena and acoustic variation on interjections, pauses and repairs in spontaneous speech (in Japanese). J. Acoust. Soc. Jpn. (J),51(3):202–210, 1995.
[3] A. Kai and S. Nakagawa. Investigation on unknown word
processing and strategies for spontaneous speech understanding. In Proc. of Eurospeech ’95, pp. 2095–2098, 1995.
[4] D. O’Shaughnessy. Recognition of hesitations in spontaneous speech. In Proc. of ICASSP 92, pp. I–521–524, 1992.
[5] F. C. M. Quimbo, T. Kawahara, and S. Doshita. Prosodic
analysis of fillers and self-repair in Japanese speech. In
Proc. of ICSLP 98, 1998.
[6] Y. Takubo. Towards a linguistic model of speech performance (in Japanese). Journal of Information Processing
Society of Japan, 36(11):1020–1026, 1995.
[7] R. L. Rose. The communicative value of filled pauses inspontaneous speech. PhD thesis, University of Birmingham, 1998.
[8] J. L. Flanagan and R. M. Golden. Phase vocoder. The Bell
System Technical Journal, 45:1493–1509, 1966.
[9] F. J. Charpentier. Pitch detection using the short-term phase
spectrum. In Proc. of ICASSP 86, pp. 113–116, 1986.
[10] T. Abe, T. Kobayashi, and S. Imai. The IF spectrogram: a
new spectral representation. In Proc. of ASVA 97, pp. 423–
430, 1997.
[11] H. Kawahara, H. Katayose, R. D. Patterson, andA. de Cheveigné. Highly accurate F0 extraction using instantaneous frequencies (in Japanese). Tech. Com. Psycho.
Physio., Acoust. Soc. of Japan, H-98-116, pp. 31–38, 1998.
[12] K. Itou, T. Akiba, O. Hasegawa, S. Hayamizu, andK. Tanaka. A Japanese spontaneous speech corpus collected using automatically inferencing Wizard of OZ system. J. Acoust. Soc. Jpn. (E), 20(3), 1999.

Page 230