Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST 2005), pp.35-36,
October 2005. (Demos)

Speech Interface Exploiting Intentionally-Controlled
Nonverbal Speech Information
Masataka Goto
National Institute of Advanced Industrial
Science and Technology (AIST).
Ibaraki 305-8568, Japanm.goto@aist.go.jp

Katunobu Itou
Nagoya University.
Aichi 464-8603, Japan
itou@is.nagoya-u.ac.jp

ABSTRACTThis paper describes our research on speech interfaces using
nonverbal speech information. Although speech informationconsists of verbal and nonverbal information, most speechrecognition research has made use of only verbal information such as words and sentences. From among nonverbal
information, we have focused on hesitation (filled pause) andprosody (voice pitch) to create four speech-interface functions: Speech Completion, Speech Shift, Speech Starter, andSpeech Spotter. Hesitation, for example, can be used as atrigger to complete an uttered fragment and pitch changing
can be used to enter a word with it having different functions. By having users intentionally utter nonverbal information according to simple rules, we have achieved interfacesthat can exploit the potential of speech in various forms.
ACM Classification: H5.2 [Information interfaces and pre-sentation]: User Interfaces. - Voice I/O.
General Terms: Design, Human Factors
Keywords: Speech interface, speech recognition, nonverbal

information, filled pause, voice pitch
1 INTRODUCTION

Most speech recognition research has focused on ways of obtaining verbal information such as phonemes and words fromspeech and on improving the speech-recognition rate. The
technology developed for this purpose is, of course, important, but raising the recognition rate by itself is insufficient —interfaces using speech recognition are still difficult to use.
With the goal of using speech recognition through a comfortable and easy-to-use interface, we set out to achieve speechinterface functions that can tap the potential of speech bymaking full use of nonverbal information, which has mostly
been ignored by speech recognizers.
Nonverbal information such as hesitation and voice pitch has,
if anything, been considered a problem in that it can cause
recognition errors. For example, erroneous recognition andinappropriate input caused by hesitation during speech input is a common occurrence. Although some attempts havebeen made at using voice pitch (or prosody), mainly to improve speech recognition rates [1, 2] or to analyze phraseboundaries [3], these attempts dealt with auxiliary aspects

Copyright held by author/owner
UIST’05, October 23–26, 2005, Seattle, Washington, USA.
UIST 2005 Companion

Tetsunori Kobayashi
Waseda University.
Tokyo 169-8555, Japankoba@waseda.jp

of nonverbal information unintentionally contained in natural speech input. Although there were a few exceptions [4, 5]
that used nonverbal information for interface functions, they
did not exploit both verbal and nonverbal information.
We developed the four speech-recognition-based speechinterface functions shown in Figure 1 by using two kindsof nonverbal information, a filled pause (vowel-lengthening
hesitation)1 and voice pitch (fundamental frequency, F0),both of which are unique to speech, having no counterpartsin writing. In contrast to conventional speech input, whichconveys only verbal information, our functions place importance on intentional hesitation or pitch changing by the user
during speech input. This has made it possible to convey newtypes of information from the user side to the computer sideand to achieve interface functions that exploit features uniqueto speech.
2 SPEECH COMPLETION: On-Demand Completion Assistance Using Filled Pauses

Speech completion is a function that helps a user enter a wordor phrase by completing (filling in the rest of) a phrase fragment uttered by the user, as depicted in Figure 1. Although
the concept of completion is widely used in text-based interfaces [6], there have been no reports of completion being
effectively applied to speech. By using a filled pause, weenable a user to effortlessly invoke the speech-completionfunction, which helps the user recall uncertain (difficult toremember) phrases and saves effort when the input phrase islong. When a user hesitates by lengthening a vowel sound(by uttering a filled pause) during a phrase (like the Japanesephrase “maikeru–”2 corresponding to “Michael, uh...” or“Michael–” in English), our system immediately displays
completion candidates whose beginnings acoustically resemble the uttered fragment (like “Michael Jackson”, “MichaelMcDonald”, etc.) so that the user can select the correct one.
Experiments with 45 subjects showed the effectiveness of
this function. [7]3 SPEECH SHIFT: Direct Speech-Input-Mode Switching
through Intentional Control of Voice Pitch

Speech shift is a function that enables a user to enter the sameword with it having different meanings (functions) by intentionally controlling the voice pitch used. Current speechinput interfaces cannot distinguish a word from the same one
1 Although there are several hesitation phenomena, we target only filled
pauses, which we here use in the meaning of prolongation of a vowel sound,like “er...” (underlining indicates a filled pause).
2 When a foreign name like “Michael Jackson” is pronounced in Japanese,
it is regularized to conform to the Japanese style: “maikeru jakuson”. InJapanese, vowel-lengthening hesitations like “maikeru–” are common.

Demos35

"Michael-"("Michael, uh...")"MichaelJackson?"

Say "save" witha normal pitch

Hesitate and say"Er..., MichaelJackson" Recognize only

Dictation mode
Voice-command mode

"What's the weather forecast
for tomorrow?""Michael Jackson" "Um, shall we ask?

Say only
Speech Completion: When
Say "save" witha user cannot remember anIgnore"Michael Jackson"a high pitch
Dictation mode
entire phrase and hesitates(intentionally raised)
Voice-command mode
while uttering it, this function
Speech Starter: This function
Speech Shift: The user can specify the prevents erroneous detection of noise byhelps the user recall it byproviding options forspeech-input mode for each utterance by applying speech recognition only after
completing the utterance.
intentionally changing the voice pitch used. an intentional hesitation (filled pause).

Er..., what's tomorrow's weather?""Clear"Speech Spotter: This function
performs speech recognition
for only high-pitched utterances
following a filled pause.

Figure 1: Four speech-interface functions exploiting nonverbal speech information. Underlining indicates a filled pause(vowel-lengthening hesitation) and overlining indicates that the pitch is intentionally raised.

with a different pitch because they recognize only verbal information. As depicted in Figure 1, the speech-shift function can distinguish an utterance with a high (shifted) pitchfrom one with a normal (low) pitch and assign them to different speech-input modes. On a voice-enabled word processor, for example, it can regard the former type utterance
as voice-command-mode input (such as file-menu and editmenu commands like “delete”) and the latter type as regular dictation-mode text input. Experiments with 20 subjects
showed that the speech-shift function is an effective, easy-touse, and labor-saving input method. [8]4 SPEECH STARTER: Noise-Robust Endpoint Detection
Using Filled Pauses

Speech starter is a function that enables noise-robust endpoint (utterance) detection for speech recognition in nonstationary noisy environments. When current speech recognizers are used in a noisy environment, a typical recognition error is caused by incorrect endpoints because their automatic detection is likely to be disturbed by non-stationary
noise. As depicted in Figure 1, the speech-starter function
enables a user to specify the beginning of each utterance withan intentional hesitation (a filled pause), which is used as atrigger to start speech recognition. Since filled pauses can bedetected robustly in a noisy environment, practical, handsfree endpoint detection without using any input device otherthan a microphone can be achieved. Experiments showedthat this function provided good performances in noisy conditions at SNRs of 0 and 10 dB. [9]tion and a music-playback system for enriching telephone
conversation. Experiments showed that the speech-spotter
function is robust and convenient enough to be used in bothface-to-face and cellular-phone conversations. [11]6 CONCLUSION

We have described four novel, easy-to-use speech interfacesthat have users intentionally utter nonverbal information. We
believe that the ability of speech to simultaneously convey
both verbal and nonverbal information is of fundamental importance, and have created four speech-interface functions
that use both of these to full effect. We found that it is possible to use nonverbal information to achieve new interfacefunctions by having users utter that information intentionally.
This differs with past approaches of using nonverbal information in which the information is unconsciously uttered as
simply a supplement to verbal information.
In the future, we plan to study the synergies between thesefour separate functions and build integrated interfaces. Future work will also include further development of our research toward diverse speech-interface functions that exploit
the great potential of speech.
ACKNOWLEDGMENTS
The authors would like to thank Koji Kitayama, Yukihiro Omoto, Satoru
Hayamizu, and Tomoyosi Akiba for their valuable discussions and collaborative research on speech interfaces.

REFERENCES

5 SPEECH SPOTTER: On-Demand Speech Recognitionin Human-to-Human Conversation

Speech spotter is a function that enables a user to enter voicecommands into a speech recognizer during the course of anatural human-to-human conversation. In the past, it has
been difficult to use automatic speech recognition in humanto-human conversation situations because it was difficult tojudge, from microphone input only, 3 whether the user wasspeaking to another person or the speech recognizer. Wesolve this problem by enabling a user to intentionally indicate whether each utterance is to be accepted (processed) by
the speech recognizer by using two kinds of nonverbal information, a filled pause and voice pitch. As depicted in
Figure 1, the speech-spotter function regards a user utterance
as a command utterance only when it is uttered with a highpitch just after a filled pause (like “er...”). Using this function, we have built two application systems: an on-demand
information system for assisting human-to-human conversa3 Our speech-spotter function can be considered a hands-free version ofdual-purpose speech [10], which uses a push-to-talk button.

36

Demos

1. A. Waibel. Prosodic knowledge sources for word hypothesization in acontinuous speech recognition system. ICASSP87, pp. 856–859, 1987.
2. A. Stolcke, et al. Modeling the prosody of hidden events for improved
word recognition. Eurospeech ’99, pp. 311–314, 1999.
3. E. Nöth, et al. VERBMOBIL: The use of prosody in the linguistic
components of a speech understanding system. IEEE Trans. on Speech
and Audio Processing, 8(5), 2000.
4. C. Schmandt. Employing voice back channels to facilitate audio document retrieval. COIS ’88, pp. 213–218, 1988.
5. T. Igarashi and J. F. Hughes. Voice as sound: Using non-verbal voice
input for interactive control. UIST ’01, pp. 155–156, 2001.
6. T. Masui. An efficient text input method for pen-based computers.
CHI’98, pp. 328–335, 1998.
7. M. Goto, et al. Speech completion: On-demand completion assistanceusing filled pauses for speech input interfaces. ICSLP 2002, pp. 1489–1492, 2002.
8. M. Goto, et al. Speech shift: Direct speech-input-mode switching
through intentional control of voice pitch. Eurospeech 2003, pp. 1201–1204, 2003.
9. K. Kitayama, et al. Speech starter: Noise-robust endpoint detection byusing filled pauses. Eurospeech 2003, pp. 1237–1240, 2003.
10. K. Lyons, et al. Augmenting conversations using dual-purpose speech.
UIST 2004, pp. 237–246, 2004.
11. M. Goto, et al. Speech spotter: On-demand speech recognition inhuman-human conversation on the telephone or in face-to-face situations. ICSLP 2004, pp. 1533–1536, 2004.

UIST 2005 Companion