Speech Shift

Direct Speech-Input-Mode Switching
through Intentional Control of Voice Pitch

Concept
New Direction of Speech Interface"save"o Exploit nonverbal speech information

with a high pitch

dictation mode
voice-command mode

• Current speech-input interfaces havenot fully exploited the potential of speech• Most speech recognizers utilize
only verbal (phoneme) information

Previous Interfaceso Word can be accepted in different modes• Problem

Make use of nonverbal speech information
intentionally controlled by a user

with nonverbal info.
(pitch and hesitation)"save"

dictation mode
enter the text of "save"?
voice-command mode
execute the voice command of "save"?

o Require explicit mode switching• Use key phrases ("dictation" / "voice command")Speech Shift

Hard to enter the phrases themselves• Use other devices (mouse or keyboard)o What is speech shift?

Awkward• Enable a user to switch speech-input modes
by intentionally changing the pitchof an utterance• Allocate two types of utterances to different modes

Voice-Enabled Word Processoro Speech-shift function enables

seamless speech-input-mode switching• normal utterance : Dictation mode
- Continuous speech dictation (text input)
timenormal utterance

shift utterancewith a low (normal) pitch with a high (shifted) pitch• Benefits
1. Switching without other devices
Can invoke functions in different modes
w/o needing to use other devices

2. Seamless switching between input modes
Can invoke functions in different modes
w/o switching between input modes explicitly
w/o needing to be aware of
the current input modeo General idea
• Useful for any voice-enabled applications• Intentional pitch control brings
additional information to speech-input interfaces"save"

with a normal pitch

dictation mode
voice-command mode

• shift utterance : Voice-command mode
- Edit-menu and format-menu commands
delete, backspace, bold, left justify, right justify,
center justify, new line (enter), undo, cut, paste, etc.
- File-menu commands
save, open (file open), print, close document, etc.

"save"

with a high pitch

dictation mode
voice-command modeo Permit filled pauses• Encourage a user to hesitate with filled pauses
Utterances with filled pauses are
not accepted as dictation

M. Goto , Y. Omoto , K. Itou , and T. Kobayashi
112212National Institute of Advanced Industrial Sci. and Tech. (AIST)
Waseda University

Implementation
Audio signal input

Filled-pause detector
Pitch (F0)F0 estimator

Filled-Pause Detector
Filled-pause
periodSpeech-shift classifier- thresholding method- command-context method

Utterance detector

N-bestrecognition
results(Endpoint detector)

Speech recognizer

Interface manager(LVCSR, CSRC toolkit)(Word processor)o Detect the beginning of each filled pause• Real-time filled-pause (FP) detection method[Goto et al. 1999]- Independent of vocabulary and language- Detect a lengthened vowel in any word

• Bottom-up acoustical analysis- Two features of filled pause (FP)Speech-Shift Classifiero Distinguish between

normal and shift utterances• Difficult to judge whether the pitch is shifted- Pitch range differs among individuals• Base fundamental frequency (base F0)

Small pitch transition- Unique pitch reference for each speaker- Estimate by averaging the voice pitch
during a filled pause (FP) (e.g., "er...")- Gradually update the base F0 for every FPExperimental Resultso Evaluation of classification performance• Relative pitch value- Pitch value relative to the base F0• Tested on 60 normal and 60 shift utterances- Uttered by 12 Japanese subjects

o Thresholding method (for general purposes)"er..."

utterance

utterance

average pitch

average pitch

97.1

94.6

"you can use the new line" "new line"

filled pause

Small spectralenvelope deformation

95.3

89.4

individual model for each subject
common model for all the subjects

timethresholding method

command-context method

Both methods are robust enoughthresholdo Usability evaluation of word processor

base F0
normal utterance

shift utterance

timeo Command-context method (for word processor)• Incorporate prior knowledge about
the linguistic context of voice commands
W: word sequence, C: command-flag sequence
X: spectrum sequence, A: pitch sequence^ ^{W, C} = argmax P(W, C | X, A)
W, C= argmax P(A | C) P(C | W) P(W | X)
W, C

results ofword-pitch model
speech recognizer
command-flag model• Tested with 20 Japanese subjects• Compare 4 input methods(a) modes are switched by mouse operation(b) switched by uttering predefined key phrases(c) commands are uttered while pressing shift key(d) commands are entered by speech-shift function• Results- Required time: (a) and (d) took the shortest- Relative usage frequency: (d) was 79.8%- Questionnaire results: (d) was most preferred,easy to use, and labor-saving- 85% of the subjects wanted to use (d) in the futureVideo clips of our speech-interface projects:
http://staff.aist.go.jp/m.goto/EUROSPEECH2003/
http://staff.aist.go.jp/m.goto/ICSLP2002/

Snapshots(1) Uttering "er...":The base F0 (the pitch of thespeaker's natural voice) is
updated whenever a filled
pause is detected.

(2) Uttering "new line"with a normal pitch:
A normal utterance is regarded as regular dictation-mode
text input.

(2) Uttering "new line"with a high pitch:
A shift utterance is regarded
as voice-command-mode input.

Summaryo Propose a new speech interface function"Speech Shift"• Make use of nonverbal speech info. (pitch)• High-pitch voice has a good "Shift" function- Pitch of natural voice is estimated by using FPs- Naturally used in ventriloquism• Effective means of entering voice commands• Can be applied to various speech applications

Future Directionso Interfaces using intentional nonverbal info.
• "Speech Completion" [HCI Intl. 2001] [ICSLP 2002]• "Speech Shift" [Eurospeech 2003]• "Speech Starter" [Eurospeech 2003]• "Speech ???"
Further developing this concept...
2003/09/02 Eurospeech 2003 poster