THE USE OF ACOUSTICALLY DETECTED FILLED AND SILENT PAUSES INSPONTANEOUS SPEECH RECOGNITION
Jun Ogata1 , Masataka Goto1 , and Katunobu Itou2
1

National Institute of Advanced Industrial Science and Technology (AIST).
Ibaraki 305-8568, Japan
2
Hosei University. Tokyo 102-8160, Japan

ABSTRACTIn recognizing spontaneous speech, the performance of typicalspeech recognizers tends to be degraded by ﬁlled and silent pauses,
which are hesitation phenomena frequently occurred in such speech.
In this paper, we present a method for improving the performanceof a speech recognizer by detecting and handling both ﬁlled pauses(lengthened vowels) and silent (unﬁlled) pauses. Our method automatically detects these pauses by using a bottom-up acoustical analysis in parallel with a typical speech decoding process, and thenincorporates the detected results into the decoding process. From
the results of experiments conducted using the CIAIR spontaneous
speech corpus, the effectiveness of the proposed method was conﬁrmed.
Index Terms: spontaneous speech, ﬁlled pause, silent pause, acoustic model, language model
1. INTRODUCTION
Current state-of-the-art speech recognition systems can achieve highrecognition accuracy for read speech or spoken dialogue in limited
domains. However, the recognition of spontaneous speech still remains problematic because such speech includes various phenomena such as ﬁlled and silent pauses, repairs, hesitations, repetitions, and partial words. To improve the accuracy for spontaneous
speech, novel improvement techniques to reduce the recognition errors caused by such phenomena are required.
As a ﬁrst step toward dealing with such natural phenomena inspeech recognizers, we focus on two important phenomena, namely,
ﬁlled pauses (lengthened vowels) and silent (unﬁlled) pauses. Whileﬁlled and silent pauses play an important role in spoken language,
for example, in helping a speaker hold a conversational turn and express mental and thinking states [1][2], these pauses tend to cause
recognition errors in typical speech recognition systems. For example, a Japanese sentence “ee- washoku- no resuto-ran’ e ...” (English translation: “er, to a Japanese food restaurant ...”) includes
three ﬁlled pauses (lengthened vowels), which are represented by“-” and underlines. Note that in this paper we use the term “ﬁlled
pause” for a vowel-lengthening phenomenon such as a ﬁller (e.g.,“ee-”, “maa-”, and “ano-” in Japanese) and a lengthened vowel during a word. In the above Japanese sentence, three types of ﬁlled
pauses appear (1) as a ﬁller “ee-” (“er” in English) (2) at the endof a word “washoku-” (“Japanese food” in English), and (3) withina word “resuto-ran” (“restaurant” in English). Although acousticalproperties of ﬁlled pauses are common in most languages, their probable positions in an utterance depend on languages — for example,
the latter two types, (2) and (3), are quite popular in Japanese spontaneous speech. On the other hand, the term “silent pause” basicallymeans a temporal region in which a speaker does not utter during a

978-1-4244-2354-5/09/$25.00 ©2009 IEEE

4305

word, phrase, or sentence in spontaneous speech. In particular, during spontaneous conversations (at least in Japanese), relatively longsilent pauses tend to be inserted at any positions in an utterance, bothwithin a word as well as between words.
Several previous methods have dealt with ﬁlled pauses in speech
recognition. In [3], a language model that considers speech disﬂuency was presented. In [4], ﬁlled pauses were incorporated into pronunciation variations for lexical modeling. Most of these approaches
assumed that ﬁlled pauses can be dealt with as words in a system
vocabulary and that a speech recognizer can reliably generate hypotheses containing such pauses as words. However, Stouten et al.
[5] reported that this assumption is often wrong because it is difﬁcult to predict the occurrences of ﬁlled pauses by using acoustic,language, and pronunciation models (lexicons). Therefore, they proposed a method that uses an independent ﬁlled-pause detector to reduce speech recognition errors caused by disﬂuencies. This method[5] ﬁrst segments the input speech into phoneme-like segments, andthen judges whether each segment is a ﬁlled pause or not (i.e., detects a ﬁlled pause) by using a neural network before speech recognition. All the frames of the detected ﬁlled-pause segments are thenignored when decoding, resulting in the avoidance of recognition errors caused by ﬁlled pauses. Although this method is effective forﬁllers such as “uh” and “uhm”, it did not take into account a ﬁlled
pause at the end of a word or within a word, which we deal with byusing a bottom-up acoustical analysis without segmenting the input
speech. Moreover, this method did not deal with silent pauses at all.
In this paper, we present a robust recognition method that candeal with both ﬁlled and silent pauses in spontaneous speech inJapanese. First, two detectors, one for ﬁlled pauses and the otherfor silent pauses, identify the beginning and end times of each pause
in the input speech. Then the detected pauses are utilized to control the decoding process in our speech recognizer. Since both pause
detectors are based on a bottom-up acoustical signal analysis, ourmethod has two advantages:• It can handle all types of ﬁlled pauses, regardless of where aﬁlled pause occurs in utterances — i.e., as a ﬁller (between
words), at the end of a word, or within a word.
• It does not depend on training data, task domains, and languages.
In the following sections, we ﬁrst present an overview of our method
and explain the ﬁlled and silent pause detectors. We then present a
decoding method using the detection results. Finally, we present
the results of several experiments using a large-scale spontaneous
speech database and conﬁrm the effectiveness of our method.

ICASSP 2009

Input speech

1. Small F0 (fundamental frequency) transition
When the tension of the vocal cords is unvaried under constantarticulator parameters, the F0 of the voice remains almost constant.

Training data

Filled- and silent-pause detection

2. Small spectral envelope deformation
When the vocal tract shape is unvaried under constant articulator parameters, the spectral envelope forming the formants remains almost constant. When the deformation of the envelope
is evaluated, it is necessary to eliminate the air ﬂow’s amplitude
modulation, since the air ﬂow from the lungs may vary.

FilledFilled- and silentsilent-pause
pause
regions

Removing pause regions
Language
model

Decoding withpause skipping

Acoustic
model

Training data
Re-train ing

without pauses

The method determines the beginning and end times of each ﬁlled
pause by ﬁnding the above two acoustical features of ﬁlled pauses.
Experimental results for a Japanese spontaneous speech corpus
showed that this method can detect, in real time, ﬁlled pauses with arecall rate of 84.9% and a precision rate of 91.5%. For more details,see [2].

Recognition results
Fig. 1. System overview.
2. SPEECH RECOGNITION BASED ON FILLED AND
SILENT PAUSE DETECTION
The basic concept of our method is to control a decoding process
on the basis of the output of ﬁlled and silent pause detectors. In our
method, this modiﬁed decoding based on detected pauses is carriedout only for utterances including detected pauses, and normal decoding is carried out for the other normal utterances. In this manner,
for normal utterances, we can minimize the recognition errors thatmight be caused by our modiﬁed decoding.

2.3. Silent pause detection method
For silent pause detection, we can apply several techniques that wereoriginally developed for audio classiﬁcation and segmentation tasks[6]. In this work, we employ an energy-based speech/silence classiﬁer, i.e., silent pauses are detected based solely on the log energy of
the input speech signals.
2.4. Decoding method with pause skipping

2.1. System overview
Figure 1 shows the overview of our recognizer. The ﬁrst step is to
use the ﬁlled and silent pause detectors to identify the beginning andend times of ﬁlled and silent pauses in the audio signals of the input
speech. Then, a modiﬁed decoding by skipping the detected pauses
is conducted to obtain speech recognition results. However, sincesuch a skipping process implies the intentinal manipulation (shortening) of input utterances, it might cause an acoustic mismatch between the testing and training data. In order to reduce the acoustic
mismatch and incorporate the acoustic changes caused by the skipping process, we introduce the re-training of acoustic models by using the pause detection. Even in the training data of the acoustic
models, our system automatically detects ﬁlled and silent pauses ina similar manner. By using the beginning and end times of these
pauses, it removes these temporal regions from the acoustic feature
sequences (e.g., MFCCs) of the training data. The acoustic modelis then re-trained on these sequences without the detected ﬁlled andsilent pauses.
2.2. Filled pause detection method
To detect ﬁlled pauses, we use a real-time detection method proposedby Goto et al. [2]. The basic idea of this method is to ﬁnd acousticalfeatures of ﬁlled pauses in speech signals by using frequency analysis. If ﬁlled pauses (lengthened vowels) are uttered while the speaking process is waiting for the next speech content from the thinking
process, a speaker cannot change the articulator parameters during
the ﬁlled pauses because subsequent utterances have not yet been
prepared. Hence, the method assumes that a ﬁlled pause contains acontinuous voiced sound of an unvaried phoneme, because such asound is uttered when the vocal cords are vibrated with almost constant articulator parameters (i.e., with a constant vocal-tract shape).
This method accordingly detects ﬁlled pauses on the basis of the following two features:

4306

We introduce a pause skipping mechanism into a typical timesynchronous Viterbi search (baseline recognizer), as shown in Figure
2. When the search process arrives at the beginning time (frame) of
a detected pause, all the nodes (HMM states) at this frame are maintained (stored). The search process does not decode anything at eachframe during the pause region — i.e., each detected pause is skipped
in decoding. Finally the search process is resumed from the end timeof the pause region by using the maintained node information.
Note that in the case of silent pauses, an extra operation is required in the decoding process. If all the frames of a silent pause
between words are skipped (discarded) completely, the identiﬁcation of the word boundary in the decoder becomes very difﬁcult, andthis may lead to more recognition errors than the baseline recognizer.
Therefore, in this work, some frames (i.e., a short silent region) are
not discarded but retained for the decoding process. In more detail,the length of each detected silent pause is shortened to a ﬁxed length(0.1 sec. in this paper).
This decoding method might be considered almost equivalent
to a method that ﬁrst removes the temporal regions of pauses directly from the acoustic feature sequences (MFCCs) and then performes recognition by using a regular recognizer without any extension. However, direct handling of the pause information during thedecoding process has the advantage of affording a variety of possibilities for further improvements and extensions such as incorporating
linguistic context information around each type of pauses into thedecoding process, and dynamically controlling decoding parameters(language model scaling factors, word insertion penalties, etc.) in
the decoding process.
3. EXPERIMENTS
To investigate the effectiveness of our method, we conducted recognition experiments with actual spontaneous speech data.

Filled pause

Silent pause

time

Skipping pause region

All the nodes (HMM states)
at this frame are maintained

Search process is resumed
based on the maintained nodes

Fig. 2. Decoding with pause skipping.
3.1. Database
We evaluated our method on the CIAIR in-car speech database [7],which includes a large amount of speech data collected in the formof real-world spoken dialogues in a car. This database contains multichannel data recorded from 16 microphones that are placed in various positions, and we used the speech data recorded at the “driver
headset” position. Note that this speech data is relatively clean ascompared to that from the other channels. Filled and silent pauses
occur frequently in the speech data because the CIAIR database consists of utterances of car-navigation dialogues that were recorded
while each speaker was driving an actual car in a town. In our evaluation, since utterances in which our ﬁlled-pause detector does notdetect ﬁlled pauses are simply recognized by a regular recognizer
without any extension, we excerpted test-set utterances including
ﬁlled and silent pauses from the CIAIR database (11190 utterancesby 101 speakers) as follows. First, the ﬁlled-pause detection was
performed with all the utterances in the data set. By doing so, ﬁlled
pauses were detected for 1658 utterances, and therefore, we set aside
these utterances as the test set. Note that these 1658 utterances also
include silent pauses.
3.2. Baseline recognizer
The baseline recognizer was constructed on the CIAIR database. For
the training data of acoustic modeling, we used 79093 utterances by
401 drivers. The speech recognizer uses a Japanese syllable-based
HMM [9] that consists of 245 acoustic units. This acoustic model
has no context dependencies across the acoustic units (i.e., monosyllable). For the training data of the language model, we used thetranscriptions of the spoken dialogue (94306 sentences). A bigram
language model was trained on this text data. The vocabulary size
was 6528.
The conventional way of handling pauses was incorporated in
the baseline language model. With regard to handling ﬁlled pauses,
Japanese ﬁllers were modeled in the language model by using transcriptions from the CIAIR. On the other hand, word-end silent
pauses were modeled in the lexicon by adding a short-pause (SP)phone at the end of every pronunciation.
3.3. Experimental results and discussion
The experimental results of our proposed method are summarized in
Figure 3. In this ﬁgure, “Baseline” indicates the word accuracy of thebaseline recognizer, “FP” (ﬁlled pause) and “SP” (silent pause) indicate the word accuracy of the decoding method wherein each type ofthe detected pauses was incorporated independently, and “FP+SP”(ﬁlled and silent pauses) indicates the word accuracy of the decoding method wherein both types of the detected pauses were incorpo-

4307

Fig. 3. Performance improvement of our proposed method (FP +SP).
rated. We also show the recognition performance in which the acoustic model was re-trained by using the training data after skipping thedetected pauses. In the following sections, we show the evaluationof each method individually and discuss the performance.
3.3.1. Evaluation of recognition with ﬁlled-pause detection
First, we compare the recognition performance using the ﬁlled-pause
detector (FP) with the baseline recognition (Baseline). As shown in
Figure 3, the word accuracy was improved (71.15% ⇒ 73.01%), andit was found that ﬁlled-pause detection is helpful for spontaneous
speech recognition. In addition, acoustic-model re-training achieved
a further improvement (73.01% ⇒ 73.61%). By investigating therecognition results in detail, we found that larger improvements wereobtained for word-end ﬁlled pauses (e.g., “washoku-”) and wordinternal ﬁlled pauses (e.g., “resuto-ran”).
In spoken dialogue tasks such as those in this experiment, ﬁlled
pauses tend to occur in the case of any proper nouns (e.g., shop name,place name, and product name) uttered by a speaker. Therefore, it
is very difﬁcult to expect the occurrences of such ﬁlled pauses evenif we train a language model or a lexical model. Furthermore, ourmethod could suppress insertion errors for ﬁller words that have alonger duration (e.g., “ee-” and “ano-”).
3.3.2. Evaluation of recognition with silent-pause detection
Next, we compare the performance of the recognition using thesilent-pause detector (SP) with the baseline recognition (Baseline).
As shown in Figure 3, the word accuracy was improved (71.15%⇒ 72.48%) by the silent-pause detection. By using the proposed
method, we conﬁrmed larger improvements, especially for withinword silent pauses. The following are examples of utterances including a silent pause (denoted as “[sp]”) that were not correctly
recognized by the baseline recognizer and resulted in two or threeincorrect word fragments:• Utterance where “hoteru” (“hotel” in English) was uttered as “ho[sp] teru”• Utterance where “imaikeshiten’” (“Imaike branch” in English)
was uttered as “imai [sp] keshiten’”Our method was able to correctly recognize both of these examples (i.e., the performance was improved in these cases). In a real

Example 1•Transcript•Utterance•Baseline•Proposed

Example 2•Transcript•Utterance•Baseline•Proposed

4. CONCLUSIONS(``I wanna go to a gas station …’’ in English)： gasorin’sutan’do ni iki tai： ga [sp] sorin’ sutan’do ni iki tai： ga sou ni sutan’do ni iki tai： gasorin’sutan’do ni iki tai(`` … to the palace of …’’ in English)： no kaijou e： no [sp] kai [sp] jo [sp]： dokka ii ji dou itte
： no kaijou eeFig. 4. Examples of improvements obtained by our proposed method.
“Transcript” shows a correct word sequence, “Utterance” shows anactual utterance with ﬁlled pauses (“-” with underlines) and silent
pauses [sp], “Baseline” shows a recognition result by the baseline
method without any extension, and “Proposed” shows a recognition
result obtained by our proposed method by skipping both ﬁlled andsilent pauses.
world spoken dialogue, since a speaker tends to utter while thinking,such silent pauses occur even within a word. In particular, “syllablestressed speech” [8] that occurs in making error recovery utterances
tends to include a silent pause within a word. It is therefore important to improve the performance for such utterances.
Note that it was generally difﬁcult to improve the performanceon a silent pause between words because the lexicon used in thebaseline language model has already dealt with such a silent pause.
When noise sounds such as breath, tongue clicking, and small incar noise were observed during long silent pauses between words,
however, they caused some insertion errors with the baseline recognition. Our method with the energy-based detector correctly ignored
most of these small noise sounds below a threshold and improved
the performance.

In this paper we presented a decoding method that can reduce
recognition errors caused by ﬁlled and silent pauses in spontaneous
speech. These two types of pauses are detected independently byusing their acoustical features, and the detection results are used tocontrol the decoding process so that it can skip such pauses. Our proposed method achieved signiﬁcant improvements on a spontaneous
dialogue speech corpus.
Future work will include improvements in our decoding method
and further evaluations using other tasks or corpora. Since we useda context-independent acoustic model (syllable model) in the experiments, we will extend the decoding method to handle a contextdependent acoustic model, which would result in a higher performance. Furthermore, we plan to incorporate the decoding method
into our public web service, “PodCastle” [10][11], that provides
full-text searching of podcasts on the basis of automatic speech
recognition. Since spontaneous speech in podcasts covers a wider
variety of speech data and often includes ﬁlled and silent pauses, thedecoding method proposed in this paper must be effective.
5. ACKNOWLEDGEMENTS
This work was partially supported by Grant-in-Aid for Scientiﬁc Research (B) (KAKENHI 19300065).
6. REFERENCES[1] E. Sheriberg: “To ‘errrr’ is human: ecology and acoustics ofspeech disﬂuencies”, Journal of the International Phonetic Association, vol.31, no.1,pp.153-169, 2001.
[2] M. Goto, K. Itou, and S. Hayamizu: “A real-time ﬁlled pause
detection system for Spontaneous Speech Recognition”, In
Proc. of Eurospeech 1999, pp.227-230, 1999.
[3] A. Stolcke and E. Shriberg: “Statistical language modeling forspeech disﬂuency”, In Proc. of ICASSP ’96, pp.405-408, 1996.

3.3.3. Evaluation of recognition with ﬁlled and silent-pause detectors
Finally, we describe the recognition results of the ﬁlled and silent
pause detectors (FP+SP). As shown in Figure 3, the improvements
obtained by using both detectors together were better than those obtained by using each detector alone. As compared to the baseline
recognition (Baseline), it eventually increased the word accuracy
from 71.15% to 74.73%. This suggests that both ﬁlled and silent
pauses actually occurred in spontaneous speech or dialogues suchas those in the CIAIR database. Both ﬁlled and silent pauses cansometimes occur within just one word. Figure 4 shows examples ofsuch utterances that were actually observed in our experiment. In
Example 1, an utterance of “gasorin’stan’do”(“gas station” in English) includes a silent pause “ga [sp] sori” and a ﬁlled pause “n’-”within this word. In Example 2, an utterance of “kaijou” (“venue” inEnglish) includes a silent pause “kai [sp] jo-” within this word anda ﬁlled pause “jo-” at the end of this word. In addition, two othersilent pauses also occurred before and after the word “kaijou”. As
shown in this ﬁgure, the results of the baseline recognition included
insertion errors, but those of our proposed method showed signiﬁcant improvements for these difﬁcult utterances. We thus conﬁrmed
the effectiveness of dealing with both ﬁlled and silent pauses in the
decoding process.

4308[4] H. Schamm, et al.: “Filled pause modeling for medical transcriptions”, In Proc. of SSPR, pp.143-146, 2003.
[5] F. Stouten, et al.: “Coping with disﬂuencies in spontaneous
speech recognition: Acoustic detection and linguistic context
manipulation”, Speech Communication, 48, 1590-1606, 2006.
[6] D. Wang, et al.: “Speech segmentation without speech recognition”, In Proc. of ICASSP 2003, pp.468-471, 2003.
[7] K. Takeda, et al.: “Construction and Evaluation of a Large incar Speech Corpus,” IEICE Transactions on Information and
Systems, Vol. E88-D, No.3, pp.553-561, 2005.
[8] K. Okuda, T. Matsui, S. Nakamura: “Toward the creation of
acoustic models for stressed Japanese speech”, In Proc. of Eurospeech 2001, pp.1653-1656, 2001.
[9] J. Ogata and Y. Ariki: “Syllable-based acoustic modeling forJapanese spontaneous speech recognition”, In Proc. of Eurospeech 2003, pp.2513-2516, 2003.
[10] M. Goto, J. Ogata, and K. Eto: “PodCastle: A web 2.0 approach to speech recognition research”, In Proc. of Interspeech
2007, pp.2397-2400, 2007.
[11] J. Ogata, M. Goto, and K. Eto: “Automatic transcription fora web 2.0 service to search podcasts”, In Proc. of Interspeech
2007, pp.2617-2620, 2007.