An Analysis of Human-to-Human Dialogs andIts Application to Assist Visually-Impaired
PeopleTakuya Nishimotoa) and Takayuki Watanabeb)a)

Graduate School of Information Science and Technology, The University of Tokyo
7-3-1, Hongo, Bunkyo-ku, Tokyo, 113-8656, JAPANb)
Department of Communication, Tokyo Woman’s Christian University
2-6-1 Zenpukuzi, Suginami-ku, Tokyo, 167-8585, JAPANnishi@hil.t.u-tokyo.ac.jp
nabe@lab.twcu.ac.jp
http://hil.t.u-tokyo.ac.jp/∼nishi/

Abstract. A prototype lunch delivery web system for visually impaired
was developed based on the analysis of human (a visually impaired whowants to order a lunch) to human (a sighted person who helps the visually
impaired to decide lunch by reading aloud lunch menu) dialog. Based on
the analysis, a prototype system was developed, which consists of threesteps: 1) rough selection, 2) selection of favorites, and 3) ﬁnal selection.
The evaluation of the prototype system is performed by the visually
impaired only with synthesized speech and key-board. The results showedthat our system is eﬀective and easy to use.
Key words: exploratory search, web application, dialog system, screen
reader1Introduction

People with visual disability access the web with screen readers. They listen to
information output in synthesized speech and operate computer using key-board.
However, it is not easy for them to carry out the exploratory search[1]. There aremany candidates in exploratory search and the comparisons are very important.
In such task, we cannot know the operations or the keywords to choose a certain
item eﬃciently. Such activities are necessities in our daily life.
Supposing that we choose a meal which a person wants to eat at the restaurant, we read the menu at there. Various information written on the menu includes the names of the meals, prices, detailed illustrations, and the pictures.
The meal which the person wants to eat is not always same, and it changes byfeeling and physical condition. In case our desired meal is not included in themenu, we must look for another choice. While repeating search and making decision, we gradually change a vague desire to the speciﬁc request. The general
requirements in such exploratory search are as follows; 1) The user wants to2Takuya Nishimoto, Takayuki Watanabe

Fig. 1. A part of the menu of lunch delivery service. Pictures, names of the items,
prices are shown.

understand entire of the menu roughly. 2) With the time and eﬀort as little aspossible, a choice must be made that he or she can be satisﬁed with.
In this study, we start with the observation of the behavior of expert supporter who satisfy the request of people with visual disability in such a problem,
continue with the investigation of the factors that the expert takes into consideration. Finally we show an implementation of a web application for people withvisual disability and evaluate the application with a exploratory search task.

2

Human-to-Human Dialog Experiment

Two dialogs by two totally visually impaired persons and one supporter wererecorded. The supporter was a woman with the volunteer experience of ten years.
The task was to choose a lunch which the subject wanted to eat in considerationof his mood and the physical condition from the menu of a lunch delivery service(Fig. 1). The dialogs of 20 to 40 minutes were performed. The dialogs weretranscribed and analyzed manually.
As the result of our analysis, eﬀective communication was observed in thedialogs between human and human to ﬁnd the relevant information in a shorttime. In addition, consideration to reduce the mental workload of the visually
impaired was accomplished.
To let the machine act as the human assistant, we created a manual for theautomated conversation. The menu was reorganized so that the user can understand it easily. Furthermore, the function for memorizing the search conditions
and the favorite items, which was similar to the so-called shopping cart in many
web shopping sites, was introduced. Only the number of candidates which meetthe conditions was shown in the stage of the rough selection, and detailed information of the favorite items was shown for the comparisons and the ﬁnaldecisions.
We performed the Wizard of Oz experiments to test our interaction design.
Although it was the system-initiative conversation, eﬀective communications

An Analysis of Human-to-Human Dialogs to Assist Visually-Impaired People

3Fig. 2. Screen images of the system; (a) ”Do you want to listen to explanation of thewhole menu?” (b) ”Items are divided into 3 groups by price. Please check the items
which you want to leave as candidates.” (c) ”54 items are selected. You can narrow
down the candidates using conditions on price and contents. Do you add a condition?”were realized. The users of the system, who were sighted university students,
reported that they felt the conversations were very natural.

3Prototype System and Evaluations

A prototype web application was implemented based on the manual which wasmade for the Wizard of Oz experiments(Fig. 2). The system consists of threesteps: 1) rough selection (candidate items are roughly selected based on categories), 2) selection of favorites (favorite items are selected from candidate items
and stored in the system), and 3) ﬁnal selection (one item is selected with detailed information for each item). To generate XHTML document dynamically,
PHP and PostgreSQL were used.
Two persons with visual disability used the application with the screen reader
for the evaluation. First subject used the JAWS 6.2 (Japanese version), andsecond subject used the IBM Home Page Reader 3.02 (Japanese version). Only
the synthesized speech and key-board were used.
During the ﬁrst experiment, Undo command, which cancels the previous
operation, was prohibited, because it was diﬃcult to keep consistency with thesession management. In case the Undo command cannot be used, however, wefound that the user took the conservative strategy and he didn’t use the functions
which may reduce the time for ﬁnishing the task. The Undo command wasimplemented after the ﬁrst experiment.
The second subject utilized the rough selection and the favorite item list,and ﬁnally, the reasonable selection was performed eﬃciently. The function tomemorize favorite item served to reduce mental workload of the user. Information
was given very eﬃciently because the detailed information was given only after
the item was registered in the favorite list. The amount time was 8 minutes. The
detailed procedures are as follows:
1. (Step 1) “select by category name” was selected

4

Takuya Nishimoto, Takayuki Watanabe

2. narrowed down from 73 to 12 items using conditions
3. without detailed information, 8 items were stored to the favorites
4. (Step 2) detailed information of the 8 items were read aloud
5. 6 items of the favorites were remained
6. (Step 3) detailed information of the 6 items were read aloud repeatedly
7. 1 item was ﬁnally selected
As the result, the system was judged to be easy to use in comparison withthe existing web sites of lunch delivery shops.
The built-in functions of the screen reader were also used eﬀectively by thevisually impaired subjects. For example, according to a subject, the operation
to listen to information which was read just before was more comfortable toperform than to ask the partner to read something again by voice. In addition,
there was a comment that the speech recognition was not necessary for this kindof system, because it doesn’t seem convenient to operate this system by speech
recognition.

4Conclusions

The results of this research suggested that the conversations between humanbeings have the rational structures. Natural interaction can be realized in theweb environment by adopting this structure. The speech recognition was notnecessary for the natural interactions. The expert supporters have the specialtechnique; they can take consideration to reduce cognitive load of the visually
impaired persons, they can make up for visual information, and they can makeverbalization of information clearly. As the future works, this technique must beapplied to more general tasks. The eﬀective usage of multi-modal interface mustalso be examined.

Acknowledgments
This work was partially supported by the Ministry of Education, Culture, Sports,
Science and Technology (Japan), Grant-in-Aid for Scientiﬁc Research on Priority
Areas, #16091210, 2004. We would like to thank Kanako Ogawa, Fumi Fujiwara
and Tomoko Shimonaga for their support and assistance in this research.

References
1. Gary Marchionini: ”Exploratory search: from ﬁnding to understanding”, Communications of the ACM, Vol.49, Issue 4, pp.41-46, 2006.
2. Takuya Nishimoto, Kanako Ogawa, Takayuki Watanabe: “Analysis of Humanto-Human Dialogs to Assist Visually-Impaired People,” IEICE Technical Report,
WIT2005-75, pp.7-12, Mar 2006 (in Japanese).
3. Takuya Nishimoto, Shigeki Sagayama, Fumi Fujiwara, Tomoko Shimonaga,
Takayuki Watanabe: ”An Analysis of Human-to-Human Dialogs and Its Application
to Assist Visually-Impaired People,” IPSJ Technical Report, 2007-SLP-11, pp.55-60,
Feb 2007 (in Japanese).