Improving Human Interface in Drawing Tool
Using Speech, Mouse and Key-board
Takuya Nishimoto, Nobutoshi Shida, Tetsunori Kobayashi, Katsuhiko Shirai

Department of Electrical Engineering, Waseda University
3{4{1 Okubo, Shinjuku-ku, Tokyo 169, Japan

Abstract

This paper focuses on the utility of speech input.
We proposed some principles of humen-computer interaction, which consist from the basic principles andorganization principles of interface that are requiredfor comfortable input systems. Then applying theprinciples, we discuss the desired organization of interface using speech, mouse and key-board and designed
a multimodal drawing tool S-tgif. We also verify thebene ts of speech input using the tool. As the results,
the multimodal drawing tool with speech recognition
reduces average operation time to 82%, the numberof command inputs to 89%, and movement of mousepointer to 53% in evaluation with inexperienced users.
It is expected to make speech input more bene cial byapplying our guidelines.

1 Introduction

Speech recognition technique had greatly improvedin recent years, and many kinds of demonstration systems are realized in limited task domains[1][2]. Nevertheless, very few systems have shown the bene ts ofspeech because the alternative methods such as keyboard, mouse, and other pointing devices are often
more convenient for the task.
It comes from that the systematic investigation ofhuman-machine interface has been neglected by researchers on speech recognition.
To realize useful spoken dialogue interface systems, it is important to investigate the principles of
rapid and comfortable interface, and clarify the role ofspeech to satisfy the principles. For this purpose, theadvantages of speech input to the other input methods
must be discussed.
There are some attempts to clearify the characteristics of speech input compared to other input methods[3],[4],[5],[6],[1]. They are discussing therapidness and error rates of speech input in comparison with other input methods such as key-board andmouse. Their result are very diverse and there is little
agreement with the e ectiveness of speech input.
Though most of their approaches assume that thespeech input is the alternative way to the other input
method, Martin[7] has worked from the viewpoint thatspeech input is one of multiple modalities which cooperatively enhances user's ability. He evaluated theutility of speech input in the context of VLSI CAD

and his result supported the bene ts of speech input.
We also consider the e ectiveness of speech input
in multimodal input environment. We investigate thecharacteristics of speech input and the e ective way ofcooperation with other input methods from the viewpoint of Principles of Interface, in addition to regarding speech as an additional modality.
Many designers of interface proposed some guidefor interface design[2],[8]. Most of their methods, however, don't help us nding what functions should berealized because their approaches are lacking fundamental rules. Even in Shneiderman's Eight GoldenRules, prescriptions and principles are not separated.
In this paper, we suggest the basic principles ofinterface and the organization principles of interface,classifying the basic requirements in interface systems
and desired organizations that match various levels ofusers' skill[9]. We also investigated how speech input
contributes to satisfy these principles, and developeda drawing system called S-tgif to examine the utilityof speech input.

2 Basic Principles of Interface

From the viewpoint of human interface, we generally desire to use tools easily. This means that wewish to operate with less diculty, that we wish tolearn operations with less diculty, and that we wish
to know the permitted operations and consequenceswith ease.
We propose the basic principles of interface, what
are concerned with the required functions for comfortable input system. We call these principles \Principlesof less operation", \Principles of ease in learning operations" and \Principles of transparency of the system".
1. Principles of less operations:(a) Minimize the motion of position. The
movement of hands, nger and mousepointer should be minimized.
(b) Minimize the number of operations.
The number of operations for a commandshould be minimized.
2. Principles of ease in learning operations:(a) Make easy to remind the command
names. Names of the command should beeasy to remember. For the purpose, names
should be associated with their functions
and not be confusious with other commands.
(b) Keep consistency. All commands shouldbe operated with consistent methods within
the application and across many applications.
3. Principles of transparency of the system:(a) Make easy to perceive. The status of thesystem and available operations should beunderstood easily.
(b) Make easy to estimate results. The result produced by an operation should be always predictable.
(c) O er perfect feedback. Perfect feedback
should be always given for the operational
results.

3 Organization Principles of Interface

There are trade-o relations between basic principles.
Let us think of the operation with key-board forexample. From the viewpoint of less operation, shortcommand names are desirable and the best way ofoperation should be pressing one button of the keyboard. However, for the purpose of learning commands easily, command names should be associatedwith the names of the functions. If we meet the demand, the methods of operation would require somemeaningful words, such as names of commands in fullspelling. This kind of trade-o relations can be foundin many places.
The ratio of every principle's consideration depends
on the experiences and frequency of use, and the matter of preference. To realize the input system that iswidely accepted by various levels of users, the systemshould provide the best organization that satis es eachcondition of users.
We propose the organization principles of interface,which guides how each basic principle should be considered in organizing system to meet the demand of
every level of user.
1. Principle of supporting beginners:
Every input system for application should havecommon styles for those unfamiliar with the application or the computer system.
2. Principle of treating frequent users with priority:
Specialized usage of the application should beprovided for those who use the system frequently.
3. Principle of suggesting beginners to use the skilledusage:
Every input system should have a guidance mechanism that indicates the skilled usage for beginners.

4 Multimodal Drawing Tool Using
SpeechWe design a multimodal drawing tool using speech,mouse and key-board.
In drawing task, selecting objects and directing positions on the canvas screen are often mixed with commands changing modes such as selecting, drawing rectangles, circles, texts. Therefore, operations are apt
to be intricate with only mouse because pull-down or
pop-up menus will let the mouse pointer go away fromthe position to which the operator is paying attention.
In this case, multimodal interface using mouse, keyboard and speech input is expected to be e ective.
Applying the principles mentioned above, the following points have been considered in using speech input.
4.1

Applying Principles of Less Operations

 Changing modes with speech input is enabled,

therefore the operator can use mouse only to draw
pictures on canvas screen.

 Shortcut keys are assigned for one hand so theuser can hold mouse with another hand.

 Shortcut by speech input enables to use morecommands directly (with no tree menus) than thenumber of the buttons on key-board.

4.2

Applying Principles of Ease in Learning Operations

 The system will recognize all the words that as-sociates with the functions. To delete an object,for example, the operator would remind many
words such as \cut", \delete", \erase", \remove",etc. If the system can recognize only one word,
the operator must learn what is the acceptable
word, what is similar to learn peculiar operation.
To make easy in learning operations, the systemshould recognize as many words that the user canremind as possible.

4.3

Applying Principles of Transparencyof the System

 Speech recognition module always shows its sta-tus and recognition results, and the acceptance ofwords is also indicated with beep.

 The mode of the drawing system is shown withthe gure of the mouse pointer so that the user
can perceive the change of drawing mode without
moving the gazing point.

4.4

Applying Organization Principles ofInterface

 Speech input and other devices are available inany time so that the usage for beginners andskilled operator is permitted at the same time.

 To suggest to use abbreviations, shortcut keys

related to the speech input is displayed on thescreen.

5 Implementation

We develope a multimodal drawing system S-tgif(Speech-input tgif ) based on a public domain software.
Figure 1 is the operating style of the system. The
operator inputs commands with speech, while holding
mouse in one hand and key-board with the other hand.
The system has three modules. Figure 2 shows thewindows of each modules.
5.1

Speech Recognition ServerThe system has a speech recognition server thatruns on a general-purpose workstation. The server
communicates with the client application by the XWindow system protocols. It recognizes 76 isolated
Japanese words using Hidden Markov Models. The
calculations are processed in real-time on HP Series
735 workstation.
Available words list and recognized word is shown
independently of the tgif application window, and theshortcut keys correspond to each speech command isshown together.
The system records the motion of mouse pointerand the time required for each evaluation. The numbers of operations done by each device are counted
manually.
5.2

Rejection of Non-Command Utterances

In the situation that the speakers are allowed toutter whatever they think freely, they tend to include
some talks to someone other than the recognizer while
they talk to the recognizer. They might include sometalks to themselves also. Such utterances not directedto the recognizer make it dicult for the system torecognize the utterances of users. So it is very important to distinguish the utterances directed to thesystem from those not directed to the system.
In S-tgif, we implemente a prosody-based rejection
mechanism, which uses the distributions of the logarithm power. It would be e ective because the users
tend to speak clearly to the system than to the others,
and the direction of the speaker's face a ect the power
considerably.

We counted the required time to nish the task, themotion of mouse pointer and the number of commandoperations.

7 Results

Table 1 shows the ratio of devices that are used forcommands.
Table 1: The ratio of devices which are used for commands(%)
speech input
mousekey-boardw/speech
91
90w/o speech0100
0The e ect of using speech S is de ned by following
equation:

S=s

1 2(1)

2 1

where 1 and 2 denote the mean costs in the rst
and the second trial of group A respectively, 1 and2 are those in the rst and the second trial of group
B. This equation is based on the assumption that thee ect of using speech, the di erence in two groups, thedi erence of the tasks, and the di erence of the trial
number a ect on the costs independently.
As the results, speech inputs reduced the operation
as shown by Table 4.
Error rate of speech recognition is 7%.
Table 4: Reduction of operations(%)
operation time
mouse movement
num. of commands

8253896 Evaluation

Six subjects work on evaluation. All subjects
are university students who uses computer everyday.
They have no preliminary knowledge of tgif and S-tgif.
Evaluation is done by the following procedure.
First, we explain the outline of tgif and S-tgif andboth usage of mouse and speech input. Then subjects
practice on the system freely for fteen minutes using
speech, mouse and key-board.
The subjects are divided into two groups. In groupA, subjects drew the gure X with speech in rst trial
and draw the gure Y without speech in second. Ingroup B, they drew the gure X without speech in rst
trial and draw gure Y with speech in second trial.
In doing the tasks, precise size and layouts of theobjects are not required, but some indications are tobe obeyed about colors of some objects, size of fonts,
and width of some lines.
After the task was nished, we ask some questions
to survey the user-interface satisfaction.

8 Discussion

In this system, speech input reduced the required
operation time for a task, mouse pointer movement
and the number of commands.
The results of the questionnaire (Table5) show thesystem with speech input is bene cial especially on theeasiness of learining and reduction of diculty. It canbe concluded that the system showed the usability ofspeech input from the viewpoint of the basic principlesof interface.
As Table 1 shows, most users used speech input
though the mouse or key-board is also available. It
should be noted that operations using key-board isdicult to learn. Even for experienced user, speech
input would be convenience in small tasks like this.
In our observation, the number of cancellation commands used in the test session is very small in comparison with the number of speech recognition error. OnFigure 1: The operation style of S-tgif. The user input commands with speech, while operating key-board withleft hand and mouse with right hand.

Time(sec)
463
608
420

Table 2: Results of the experiments (group A)
1st with speech
2nd w/o speech
Movement
Num ofTime Movement
Num of(pixels)
commands (sec)(pixels)
commands
17284
3344837435
3323217
77
458
36257
56
20406
41433
35418
35

Time(sec)
944
524
295

Table 3: Results of the experiments (group B)
1st w/o speech
2nd with speech
Movement
Num ofTime Movement
Num of(pixels)
commands (sec)(pixels)
commands
44364
83
345
23422
28
41630
54
321
18194
3526207
27395
14601
44

Table 5: Results of Questionaire for Users' Satisfaction (Average, 3:neutral)
Rapidness of system response
Easiness of leaning usage
Easiness of knowing permitted operations
Easiness of handling objects on canvas
Reduction of diculty

3.8
4.3
3.33.84.0

Figure 2: An example of the screen image of the system. Left window is the client system of the drawing module.
Right window is the speech recognition server of the system.

some kind of situations in drawing application, recognition error can be recovered by inputting right command again without any undoing. This is supposed
to be the reason that the number of cancellation wasrelatively small. It may be assumed that the drawing
task is suitable for the application of speech input.

9 Conclusion

In this paper, we proposed the basic principles andorganization principles from the viewpoint of interface
usability. We investigated to apply the principles todrawing tool using speech input, then using the developed system, we veri ed the utility of speech input.
Some future works are remained such as the investigation of the usage of speech output, speaker adoptation, customizing mechanism and the integration withgaze tracking system.
Acknowledgements

This study has been conducted as a part of theProject: Humanoid at HUREL(HUmanoid REsearchLaboratory), Advancded Research Center for Scienceand Engineering, Waseda University. The authors
would like to thank ATR, NAMCO LIMITED, Nissan
Motor Co.,Ltd, and YASKAWA ELECTRIC Corp.,
for ther cooperation in this study.

References[1] Chris Schmandt, Debby Hindus, Mark S. Ackerman and Sanjay Manandher : \Observation on
Using Speech Input for Window Navigation," inHuman-Computer Interaction, D. Diaper et al.
Edit, Elsevier Science Publishers B.V. (1990).
[2] A.Rudnicky, J.Lunati and A.Franz: \Spoken language recognition in an oce management domain," Proc. ICASSP'91,pp.829-832 (1991).
[3] D.J.Cochran, M.W.Riley, L.A.Stewart : \Anevaaluation of the strengths, weaknesses and usesof voice input devices." Proceedings of the HmanFactors Society { 24th Annual Meeting (1980).

[4] J.M.Nye : \Human factors analysis of speech
recognition systems," Speech Technology, I, 5057 (1982).
[5] J.Leggett, G.Williams : \An emprical investigation of voice as an input modality for computer
programming," International Journal of ManMachine Studies, 21, 493-520 (1984).
[6] R.Haller, H.Mutschler, M.Voss : \Comparison ofof input devices for correction of typing errors inoce systems," Proceedings of INTERACT '84(1984).
[7] G.L.Martin : \The utility of speech input in usercomputer interface," Int. J. Man-Machine Studies, 30, 355-375 (1989).
[8] Ben Shneiderman: \Designing the User Interface," Second Edition, Addison-Wesley Publishing Company (1992).

[9] T.Kobayashi, Y.Takeuchi and K.Shirai: \MultiModal Input Environment Using Speech, Mouse
and Key-Board", Technical Report of IEICE,
HC92{68 (1993).