AUTOMATED TEXT TO PRESENTATION
for better understanding clinical information
PascalKUYTEN(1)

easier to be foundHelmut
PRENDINGER(2)(1)
The Universityof Tokyo, Japan

Timothy
BICKMORE(3)(2)
National Institute ofInformaticsJapan

NOVEL APPROACH FOR AUTOMATICALLY
GENERATING A VIRTUAL INSTRUCTOR FROMTEXTUAL INPUT ONLY
Our proposed system is capable of creating
virtual instruction from textual input only into
fully automated generation of agent animation
scripts from text.

Paul
PIWEK
(5)

Mitsuru
ISHIZUKA(1)(3)
NortheasternUniversity, Boston,
Massachusetts, USA(4)Columbia
University, New
York, USA

(5)
The Open
University, Milton
Keynes, UK

HIGH-LEVELDISCOURSEANALYSISCOHERENTDIALOGUE
GENERATION

text

Svetlana
STOYANCHEV(4)

rsttreeeasier to understand

EMBODIEDCONVERSATIONAL
AGENT

question-answer
pairs

scriptedanimations

virtualinstructor

SYSTEM DESIGNFirst text is translated into rhetorical structure theory(RST) trees by annotating discourse relations using
high-level discourse analysis. RST trees are thentranslated into question-answer pairs, by matchingpatters on the relations and structure of RST trees.
Answers are compiled into an animated virtual
instructor, using animation scripts.

TEXT“if you think that you have been injured by being
in this study please let the investigator know
right away”

CONDITION

HIGH-LEVEL DISCOURSE ANALYSIS

RST trees are generated by the system using a highlevel discourse analyzer, called HILDA. The discourseanalyzer first segments text into EDUs. Then, (typically)binary discourse relations are identified between EDUs.
HILDA is using three classifiers: 1) for EDUsegmentation, 2) for discourse labeling and 3) for RSTtree construction.

ATTRIBUTION
if youthinkplease let the investigatorknow right awaythat you have been injured bybeing in this study

COHERENT DIALOGUE GENERATION

Input for the CODA system is a sequence of one-level RSTtrees. It maps this to alternative (ranked) sequences ofdialogue acts, and verbalizes the top-ranked sequence.
The final output is an XML representation of a dialogue
act sequence (usually consisting mostly of questionanswer pairs).

QUESTION
What if I think that I
have been injured bybeing in this study?

ANSWER
Pleaseletinvestigatorright away.

theknow

EMBODIED CONVERSATIONAL AGENTTHEORY OF TEXT ORGANIZATION
Text can segmented into non-overlapping, semanticallyindependent units (EDUs). Between EDUs rhetorical(discourse) relations describe how the more importantpart (nucleus) and less important part (satellite) relate.
Text organization can be represented using rhetoricalstructure theory (RST) trees. Leaves represent EDUs,
arrows point from satellite to nucleus, and arrows arelabeled with a discourse relation.

THE CODA PROJECT
A parallel corpus of annotated monologues anddialogues was constructed, where the dialogues express
the same information as the aligned monologues. From
this, a mapping was inferred from RST structures inmonologue to the dialogue act sequences in dialogue.
These mapping are used by the system (CODA) to map
an RST tree to a sequence of dialogue acts.

EVALUATION

Dialogue between a single agent and a user is scriptedusing a custom hierarchical transition network-based
scripting language. Agent nonverbal conversational
behavior is generated using BEAT, and includes beat(baton) hand gestures and eyebrow raises for emphasis,gaze away behavior for signaling turn-taking, and posture
shifts to mark topic boundaries, synchronized withsynthesized speech.

VIRTUAL INSTRUCTOR
Hi, I am Karen. I am going
to explain an informed
consent document to you
for a clinical trial.

HYPOTHESISWe hypothesized that if a user conducts a question-asking
dialogue with an agent about a text, in addition to reading
the text, that they will be more cognitively engaged in thematerial, understand more about it, and be more satisfied
with the experience, compared to simply reading the text
by itself.

Our end-to-end system was evaluated with a group oftwenty-four subjects. The evaluation was conducted using
three informed consent documents of clinical trials fromthe domain of colon cancer. Each of the documents wasexplained by a virtual instructor using 1) text, 2) text andagent monologue, and 3) text and agent performing
question-answering.

QUESTION SHOWN
What if I think that I havebeen injured by being inthis study?

RESULTSKAREN RESPONCE
Please let the investigatorknow right away.

Results show that an agent explaining an informedconsent document did not provide significantly better
comprehension scores, but did score higher onsatisfaction, compared to two control conditions