Spoken Dialog System for Database Access on Internet
Takuya NISHIMOTO, Yutaka KOBAYASHI, Yasuhisa NIIMI
Kyoto Institute of Technology
Matsugazaki, Sakyo-ku, Kyoto, 606 Japanfnishi,koba,niimig@dj.kit.ac.jp
http://www-vox.dj.kit.ac.jp

Abstract
In current spoken dialog systems, the design ofuser interfaces strongly depends on tasks anddomains. We intend to develop a general purpose spoken dialog system. It extends existing
Graphical User Interface (GUI) design used in
on-line database services on the World Wide Web(WWW), so that the design of user interface
does not require special knowledge about spoken dialogs. Web documents are written in Hyper Text Markup Language (HTML) and Common Gateway Interface (CGI), which naturally
guide some dialog interactions as intended by theauthors and by the browsers. In other words,
each document has its own dialog control strategy
in itself. Making use of such strategies enables
task-oriented spoken dialogs, provided a single
set of task-independent client modules. The dialog management information is extracted frommark-up tags, accordingly free from natural languages of text. Therefore, one could even provide a cross-language feature to the system bycombining machine translation systems.

Introduction

A speech interface system is regarded as a very natural
way for unskilled people to use computers. If a system
is carefully designed to be a good combination of conventional GUI and speech interface, it also increases
the productivity.
Many standards of GUI environment are proposed,
but Motif, MS-Windows and Macintosh are now commonly used. Though many useful GUI applications
are existing now, there are few proposals to adopt isolated word recognition to existing applications. Somespeech recognition systems treat the names of icons
and menu descriptions as the voice commands, butthey use speech only as the replacement of GUI operation and do not make good use of spoken dialog.
Some strategies are important to achieve the e ective speech interface; such as \How should we organize and select the voice commands?" \How should wechoose the speech output?" and so on. Though suchproblems are task-dependent and have usually beensolved individually. But it is possible to standardize
the speech interface, if we have de nite principles touse speech.
Usually, GUI systems are created with common GUI
components such as push buttons, list-boxes, pulldown menus, etc. People can input values or con rm
processes with dialog boxes, which consists of someGUI components. Dialog boxes may be compared tothe conversations with computers. It gives a set of
information to operate step-by-step, so this set is convenient for both GUI and spoken dialog.
World Wide Web also includes, a standard of
GUI components and dialog boxes. Popular web
browsers support \forms" and common gateway interfaces (CGI). So the browser show the GUI components
and the user can access on-line databases with it.
Using CGI, web browser acts as a task-independent
application client. If we want to add speech input/output feature to some stand-alone application,
we have to make many modi cations to the original
software. On the other hand, we can add speech interface to the web applications easily. The client systemneeds some modi cations, but the web server can bepreserved. Such a system can help users who want touse speech with conventional GUI.
We propose a system that retrieves information fromthe Internet using speech input and output. This system uses only the standard protocols such as HTMLand CGI. So, we can access all the database systems
which support Hyper Text Transfer Protocol (HTTP)
and CGI protocols.
The advantages of this approach are followings:
 The client system can access to existing on-line
databases.
 The database server can provide both graphical userinterface and speech enabled interface with the same
protocols.
 The dialog control information is free from natural languages of text, because it is extracted frommark-up tags. If the system is combined with automatic language translation system, it enables crosslanguage queries.

There have been many attempts and products toapply speech input to web browsing. In many of such
systems, words on the display are treated as voice commands if the words have links to other pages. Such adesign is not so attractive as an application of speechinterface, because the usual GUI operations with amouse and a keyboard are very familiar and users
sometimes feel they are more convenient than speech.
Our goal is not only to improve the operations of
web browsers using speech interface, but also to realize a system which enables the users to do their taskeciently through natural spoken dialog.

Design Policy of Spoken Dialog Systems

A spoken dialog system is considered as the ultimate
solution to get rid of a barrier between human andmachine. It is especially useful for information retrieval
applications and on-line transactions.
There are many demonstration systems and somecommercial systems achieving such features. Most ofthem, however, ignore the presence of graphical userinterfaces, which greatly improved in these days. Most
personal computer users are accustomed to desktop
metaphors, WYSIWYG (What you see is what youget), pull-down menus and dialog boxes.
Spoken dialog systems, therefore, are not so attractive if there are no considerations to integrate speech
input/output and current GUI environment.
Recently, speech recognition systems for isolated
words have become very robust. The accuracy ofvoice command recognitions have become to satisfy thepeople's demand, but many researchers and application programmers do not create smart applications ofspeech input, so they prevent the utilization of speech
input systems.
In particular, most speech applications use voicecommands excessively, even if a mouse or a keyboard ismore convenient. On the contrary, a spoken dialog system can save the labor of users if the utility of speech
input is considered in multimodal input environment.
Many designers of interface have proposed someguide for interface design. One of the authors suggested basic principles of interface and organization
principles of interface, classifying the basic requirements in interface systems and desired organizations
that match various levels of users' skill (Nishimoto etal. 1995).
The principles consist of Basic Principles of Interface and Organization Principles of Interface.

Basic principles of interface

Basic principles of interface indicate the required functions for a comfortable input system.
1. Principles of Less Operationsa) Minimize the motion of position;
The movement of hands, ngers and a mousepointer should be minimized.

b) Minimize the number of operations;
The number of operations required for a commandshould be minimized.
c) Avoid input errors and unexpected acceptances;
The system should avoid input errors and unexpected acceptances of command.
2. Principles of Ease in Learning Operationsa) Make easy to remind the command;
Command names should be easy to remember.
For this purpose, command names should be associated with their functions and not be confusious
with other commands.
b) Keep consistency;
All commands should be operated with consistentmethods within an application and across many
applications.
3. Principles of Transparencya) Make easy to perceive the status;
The status of a system and available operations
should be understood easily.
b) Make easy to estimate results;
The result produced by an operation should bealways predictable.
c) O er perfect feedback;
Perfect feedback should be always given for operations.

Organization Principles of Interface

On which of these priciples the emphasis should bepleased depends on how long time and how frequently
the user experienced the system. To realize an input
system that is welcomed by users at various levels ofskill, a system should provide the organization which
satis es both beginners and experienced users.
Organization principles of interface guide us that
how each basic principle should be considered while
we organize a system to meet the demand of each levelof skill.
a) Principle of supporting beginners;
Every input system of application should have common styles for those who are unfamiliar with thesystem.
b) Principle of treating frequent users with priority;
Every input system should provide the specialized
usages of an application which give the experienced
users more eciency than the operations for beginners.
c) Principle of suggesting beginners to use skilled usage;
Every input system should have guidance mechanisms that indicate skilled usage to inexperienced
users.
From the viewpoint of the two sets of principles,
speech input should improve the interface, especiallyon quantity of operation.

Task Independent Design for
Multimodal Dialog System

In making a multimodal dialog system, it is importantto choose design that ts the domain of applicationand the task. For example, a multimodal drawing tool
should be designed in such a way that the mode canbe changed by speech input so that the user can use
mouse only to draw gures (Nishimoto et al. 1995).
Before describing the strategy we adopt in the spoken dialog system proposed in this paper, we will outline how to treat speech in multimodal systems so far
studied.
Usually most researchers of spoken dialog systems
decide a rst what kind of speech application is useful,then they build the systems. In many cases, the systems are well adopted for the task and useful (Smithand Hipp 1995).
The speech interface module of the system is, however, customized to the application, so it is often too
dicult to modify it to work for another application.
General purpose multimodal dialog systems havebeen also proposed, which has authoring tool or userinterface description language (Matsu'ura et al. 1994).
The application programmer who uses such systems
can create a dialog system with GUI and reusablespeech interface components. The advantage of thisapproach is that the utility of speech is fully optimized
for some task, because the design of each dialog is described individually. Suppose, for example, there be asituation where general users easily remind some word
as a voice command. If the word is not confused withthe other command words, and the operation associated with the word is easily predicative, the programmer can add the word to the voice command list.
The disadvantage is the cost of starting again fromthe interface design of existing GUI applications. At
the sacri ce of full optimization for certain application,
it is possible to build a general purpose spoken dialog
system, which extends existing GUI design and is optimized for every task to some degree. We previously described some interface principles. This approach does
not apply the principles to the whole system, but applys them to the each of the minimum components
which are used in the GUI application.

Dialog Box As A Spoken Dialog Unit

The spoken dialog system proposed in this paper uses
only the standard protocols such as HTML and CGI.
Here we will give an example of how to control a dialog
using such standard protocols.
GUI systems have a component called \dialog box."
It is a small window, including check boxes, radio buttons, text input boxes, list boxes, with the notation
which shows the meaning of values of each component.
When the system displayed a dialog box, the user has
only to nish the dialog and other operations are prohibited. The user has to pay attention to each message in the dialog box, input or modify some values,

and push OK or CANCEL button to con rm the input(Figure 1).

Font Size:
8 points
10 points

Bold

12 points

Italic18 points

Underline

24 points
CANCEL

OK

Figure 1: An example of dialog box. This dialog includes radio buttons, check buttons, and push buttons.
We treat one dialog box as one session of multimodal dialog, and apply the speech inputs and outputs to each dialog box. If there are important values,
which the user must input or make decision, the systemshould press the user for an answer. If the dialog box
contains some words which are related to the changeable values, the system adds the words to the voicecommand list (Figure 2).
SYSTEM: "Select font size"
USER: "10 points"
SYSTEM: "You can choose bold, italic or underline"
USER: "Italic and bold"
SYSTEM: "Are you OK to proceed?"
USER: "Yes"

Figure 2: An example of imaginary spoken dialog,
based on the dialog box. The system asks the user
to ll each blanks and nally con rms to the user.

Task Independent Dialog System Using
CGI

This approach can be applied to every GUI application.
But if an application is divided into the interface partand the application server, it is easy to build portabledialog management system, which accesses the same
application servers.
We use CGI protocol to realize it. Both the waysof database access and the user utterance expectation
are obtained from the mark-up les (Figure 3). The
system decides the words to recognize and the messages
to say from the texts that the form includes, and thesystem accesses to the server using the parameters in
the tags.

Dialog Management Strategies

In this section, we describe the main strategy in our
spoken dialog system.
When a dialog box is shown on the display, the dialog
system requests the user to ll the adequate values tonish the current dialog. Using the system, the user
can take the advantage of speech interface.
More than one dialogs (CGI forms) can exist in one
web page. There are also some words that are notrelated to the CGI access, but to be linked to otherpages. The system controls the dialogs between thesystem and its user according to the following strategies.

Basic Strategy

System Utterances First, the system asks the user

to select one target from the all CGI forms on thedisplay. Then, it asks the user to ll all the blanks
in the form. If all the blanks are lled, the query is
performed.
 If there are more than one forms on the screen at
the same time;
The system has multiple dialog targets. In this case,the system asks the user to choose one form as atarget.
 If the system has one dialog target (the user has
already chosen one, or the page has one form);
The system gives the order of priority to each item inthe form, based on the boldness and displayed orderof the item.
The system utterances are proceeded in the orderof priority, asking the user to ll the blank items.
Speech output is used for the purpose of making easier to understand and decreasing the time to nish
the task.
 If all the items in the form is already lled;
The system makes a con rmation, something like
\May I show you the results?" If the next utterance of the user is \Yes", it will be recognized as therequest of query.
 If the page contains only links and no dialog (form);
The system asks the user to choose one item fromthe links.
User Utterance Expectation The followings are
the utterances of the user that the system expects, andthe actions the system should take in each case.

 A word which is linked to other pages;

The system shows the destination of the link.
 A voice command which appeared in the former dialogs;
The system goes back to the status where the word
appeared, then the voice command is processed in
the dialog.
 The names of items in the form;
They are always accepted as the voice command,
whether the item is lled or still blank. Once theitem is lled by the user, the system does not ask
the user to input it again.

Strategy for Multiple Dialogs

Speech input can provide shortcut commands which
are easy to remember.
Let's assume that a sequence of dialogs be required
to nish a task. If the user wants to go backward tosome steps and correct a value, the user would repeat\cancel" operation until he/she nds the dialog which
is wanted to change. In such a case, if the speechcommands are organized properly, the same operations
can be done with one voice command. It saves the timeand labor, and it is very easy for the users to learn theoperations (Figure 4).
To realize this kind of operation, the system should
remember the keywords which appeared in former dialogs, and always accept them as the voice commands.

Priorities of the Topics

In a spoken dialog system, one dialog session has one
or more topics. The system computes the priorities ofeach topic, because if the system takes the initiativein the dialog session, the system must know in whatorder the question should be done.
In usual GUI design, important questions are written boldly or strikingly. Such items are arranged on
top of the dialog box, the size of letters is enlarged, or
the messages are prominent. Our strategy decides thepriorities of each word by analyzing such attributes of
the text shown on the display; if the size of characters on the display is large, the item is regarded as animportant item.

Case Studies

We make some investigations on some details of proposed strategies.
 If the dialog box contains the item which can notdisplay all choices at the same time (Figure 5), theuser has to push the listbox and to hold the mousebutton to see the options. In such case, if the system talks \What do you want, A, B, or C?" , theuser will know what to input next, or what options
he can choice. It meets the demand of Principles
of Less Operations and Principles of Transparency
previously mentioned.

<form><input type="radio" name="fontsize" value="8" > 8 points<input type="radio" name="fontsize" value="10"> 10 points<input type="radio" name="fontsize" value="12"> 12 points<input type="radio" name="fontsize" value="18"> 18 points<input type="radio" name="fontsize" value="24"> 24 points<input type="checkbox" name="bold"value="1"> Bold
<input type="checkbox" name="italic"value="2"> Italic<input type="checkbox" name="underline" value="3"> Underline<input type="submit" value="OK"></form>Figure 3: An example of the form written in HTML. This dialog includes 5 radio buttons and 3 check buttons. In
this case, the \name" and \value" parameters are used to access the server.
 In the case there are many choices, if the system

reads all the name of options, it takes long time andusers get tired of it (Figure 6). All the user has
to know is that the input of this item cannot be
omitted. The options themselves are easy to see on
the display. In this case, \Please read the display
and choose one option" will be more appropriate as
the system message.

 If there are many options on the display, and they

are not so essential, the system does not demand
the input. The system only accepts the items as
the voice input command. When the user wants tochoice some words, he can only to say the word, or
select by the mouse (Figure 7).

Cross-Language Extensions
Dialog management policy of the system depends onthe form tags in the original GUI system, and does notdepend on the messages itself. Therefore, the systemis free from what a language the text is written in.
There are many commercial machine translation systems aimed at web browsing. We can use such a translation system as the pre-process of the dialog management. So, if the target database system inputs or outputs messages in other language than the native one ofthe dialog system, the system needs no modi cations
in dialog management, on the assumption that the machine translation system generates accurate results.

Con guration of Proposed System
Figure 8 shows the organization of proposed system.

Machine Translation
As a language translation system, we use existing
commercial software. At present we use EnglishJapanese MT system, because our system only performs Japanese dialog management.

Dialog Target Analysis

It analyzes the HTML tags and computes the priorityof each topic as previously mentioned.

Dialog Management

It selects the item with the highest priority and asks
the user to ll the blanks using speech output. At
the same time, it also generates expectations of user
utterances.

User Event Management

It controls the speech recognition system, the speechsynthesis system, and the web browser. It manages
user events and gives multimodal feedbacks to the user.

Web Browser

User event manager sends HTML data to the webbrowser. To give the real-time feedbacks to the user,
GUI components in original HTML les are replaced
by Java applets, so they can communicate with theevent manager.

Speech Synthesis/Recognition

Speech synthesizer is a commercial text-to-speechsynthesizer. The speech recognition module uses
phoneme-based Hidden Markov Models (HMM).
Japanese words in HTML les are converted to thesequence of phonemes and used for the expectation of
the user's utterances.

Conclusion

We proposed a practical design of a general purpose spoken dialog system, and applied to the on-line
database services on the WWW. This enables taskoriented spoken dialogs with a single task-independent
client software.
As the future work, we are interested in the evaluations of the system. The dialog management module
may need some adaptations to the each user's skill, or
the reliabilities of the speech recognition.

Web Client

Database Web Site
MachineTranslation
ServerNetworkTarget Database

Database Web Site
Dialog Target
Analyzer
Target Database
HTML Page History
DialogManager

Web Browser
User EventManager

Speech Recognizer

Text To Speech

Figure 8: Organization of the system.

Display

Hidden Voice Commands

Choose Region:
EuropeNorth America
South America
AfricaAA
B
Ca) Operation with Mouse
SYSTEM: "Choose Region"
USER: "North America"A
Choose Country:
Canda
U.S.
MexicoEuropeNorth America
South America
AfricaSYSTEM: "Choose Country"
USER: "U.S."

Choose State:
AlaskaOregon....

EuropeNorth America
South America
AfricaCanadaU.S.
MexicoC

SYSTEM: "What do you want, A, B, or C?"
USER: "C"b) Operation with Spoken Dialog

Figure 5: A sample case of applying dialog strategy toa dropdown box. a) Without speech, mouse operations
are required to know the options. b) With speech, thesystem helps the user to perceive the status.

A
B
C

A
B
C

X
Y
Z

X
Y
Z

SYSTEM: "Choose State"(The user changed his mind.)
USER: "Europe, please"

Choose Country:
U.K.
France...

EuropeNorth America
South America
AfricaFigure 4: A sample case of applying dialog strategy
among pages. The user can speak the words in thedisplay (left boxes in this gure) and also hidden voicecommands (right boxes in this gure) in the former
dialogs.

SYSTEM: "Please choose one option."
USER: "C"
Figure 6: A sample case of applying dialog strategy toradio buttons. In this case, there are many options, so
the system does not read every item.

A

B

C

D

E

F

G

H

I

SYSTEM: (speak nothing)
USER: "A, F, I"
A

B

C

D

E

F

G

H

I

Figure 7: A sample case of applying dialog strategy tocheck boxes. These items are optional, so the systemspeaks nothing about them.

References

Nishimoto, T.; Shida, N.; Kobayashi, T.; Shirai, K.
1995. Improving Human Interface in Drawing Tool
Using Speech, Mouse and Key-Board. Proceedings of
4th IEEE International Workshop on Robot and Hu-: 107-112. Tokyo,
Japan.
Smith, R.W. and Hipp, D.R. 1994. Spoken Natuman Communication, ROMAN95

ral Language Dialog Systems:

A Practical Approach.

Readings, Oxford University Press.
Matsu'ura, H.; Masai, Y.; Iwasaki, J.; Tanaka, S.;
Kamio, H.; Nitta, T.; A Multimodal, Keyword-based
Spoken Dialogue System { MultiksDial. Proceedings of
ICASSP'94 II : 33-36. Adelaide, South Australia.