An analysis ofhuman-to-human dialogs
and its application toassist visually-impaired people
Takuya Nishimoto (The Univ. of Tokyo)
Takayuki Watanabe(Tokyo Woman’s Christian Univ.)

Introduction : Exploratory search• Necessary in daily life
– many candidates– comparisons are important– shortest way is not known• Not easy for visually-impaired persons– they must use TTS, instead of visual display– take longer time
– require larger mental workload• Requirements– 1) to understand the all-item list, broadly– 2) make satisfying choice; with least effort2

At a restaurant (examples)• Many items; difficult to listen to all items– at first, we should understand broadly• entree : salad, soup = 20items• main dish : beef, chicken, fish = 30items• dessert, drink = 20items– search : e.g. "exclude items with beef"• beef salad, beef steak, etc (10items are excluded)– refinement : e.g. "exclude items above 6 euro"– at last, we can compare the candidates• Names and natures often disagree...
– e.g. Sacher torte, Linzer torte– natures : chocolate cake with whipping cream3

Two human-to-human dialogs• Participants (Japanese)– two visually impaired persons– one supporter (with 10-years experience)• Task : pick out a packed lunch– from the menu of a delivery service– in consideration of moodand physical condition• Length : 20-40 min.
• Transcribed andanalyzed manually
4The skilled supporter was able to...
• Read out loud effectively– translate visual information to verbal communication• Communicate interactively– with back-channels, appropriate responses, etc.

• Lead to the relevant information– categorize items by prices, main dishes, etc.
– give easily associable nicknames– assist search refinement with conditions• Use "Shopping Cart" model– reduce the mental workload– memorize the items of interest
5Preliminary experiments• To evaluate the "shopping cart" model– a set of instructions was created– simulated human-to-machine conversations• memorize search conditions & favorite items– strategy of dialog management• rough selection : only the num of items• comparison & final decision : item details

6Examples• Single page = single task
– similar to the mobile sites• Tasks:– (a) Do you need the
explanation of the menu?
– (b) Items are divided into
three groups by price.
Please check to select.
– (c) The number of the items is 54.
Do you add the conditions for refinement?
7Implementation
Web server/ DBvoice (TTS)XHTML

VoiceBrowsing

PHP
scripts

item
information

User
keyboard

http requests(for evaluations)
JAWS / Home Page Reader

sessionhistory

undo (previous page),memorize conditions, ...
8Discussions• "Shopping Cart" model was effective– reasonable selection was made in a short time
– details are shown if the item was in the favorite list
– names of the items : essential role

• Reduction of mental workload was important– help memorizing the conditions and items• Implementation of "Undo" was important– without the function, the user was conservative– encourage to use new functions (to reduce time)• Keyboard was preferred– considered easier than using speech recognition
9

Future works• Toward the exploratory search...
– apply to other tasks– session management for "Shopping Cart"• Reconsider the architecture– session management / data model / presentation– accessibility, usability, multi-modal, i18n, security...
– ITSCJ MMI Architecture WG / W3C• Implementations (plan)– framework : Ruby on Rails– HTML : NVDA screen reader– VoiceXML : Galatea Project10