# Chronicles: Supporting Conversational Narrative in Alternative and Augmentative Communication

Annalu Waller, Rachel Menzies, Daniel Herron, Suzanne Prior, Rolf Black, Thilo Kroll

<figure><img src="image_1.png" /></figure>

## To cite this version:

Annalu Waller, Rachel Menzies, Daniel Herron, Suzanne Prior, Rolf Black, et al.. Chronicles: Supporting Conversational Narrative in Alternative and Augmentative Communication. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. pp.364-371, 10.1007/978-3-642-40480-1_23 . hal-01501812

HAL Id: hal-01501812

https://inria.hal.science/hal-01501812

Submitted on 4 Apr 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Chronicles: Supporting Conversational Narrative in
Alternative and Augmentative Communication

Annalu Waller¹, Rachel Menzies¹, Daniel Herron¹,
Suzanne Prior¹, Rolf Black¹, and Thilo Kroll²

¹ School of Computing and ²School of Nursing and Midwifery,
University of Dundee, Perth Road,
Dundee, DD1 4HN, Scotland, United Kingdom

{awaller, rachelmenzies, danielherron, sprior,
rolfblack}@computing.dundee.ac.uk;
tkroll@dundee.ac.uk

**Abstract.** Individuals share experiences and build relationships through the medium of narrative. Lifelong personal narratives play a key role in developing social identity. Individuals with little or no functional speech due to severe speech and physical impairments (SSPI) find it difficult to share personal narrative as augmentative and alternative communication (AAC) systems do not support interactive story telling. As a result, people with congenital SSPI who use AAC may not have learned the linguistic skills involved in sharing narratives. The Chronicles software was developed to support the sharing of personal narrative. Conversational analysis of a conversation using Chronicles illustrates how the system can support more natural conversations when using AAC.

**Keywords:** Augmentative and Alternative Communication, Personal narrative, Social identity, Accessibility, Assistive technology, Disability.

## 1 Introduction

People with little or no functional speech due to severe speech and physical impairments (SSPI) can benefit from augmentative and alternative communication (AAC) [1]. Although voice output communication aids (VOCAs) provide independent access to spoken output, people who use VOCAs communicate at rates of around one tenth of natural speech and find great difficulty in engaging in conversation [1]. Current VOCAs are well suited to supporting the expression of needs and wants (such as “I am thirsty”), but more complex interactions such as conversational narrative (e.g., speaking about a holiday or childhood experience) are not well supported.

The Chronicles project worked with adults with SSPI, their support staff, families and friends to harness existing research technology to support them in sharing their personal narratives. The Chronicles software was developed through ethnographic observations, interviews and rapid prototyping methodologies [2]. Researchers were
embedded in a local adult care facility during the design, development and pilot test-
ing of the software. Evaluation of *Chronicles* with an adult male with SSPI illustrates
how the system can support users in initiating and sustaining interactive narrative.

2 Background and Previous Research

Narrative has been demonstrated to shape and reflect identity and sense of personal continuity [3]. However, for many individuals who use AAC, the ability to participate in personal narrative remains severely restricted. Instead, aided conversation is characterised by one or two word responses with little initiation or elaboration [3, 4]. Most VOCAs allow for the storage and retrieval of longer units of text. However, these “stories” tend to be delivered as monologues with little opportunity for spontaneous adaptation to the natural flow of conversation. Relating personal narrative within conversation is complex and involves the retrieval, sequencing and evaluation of conversational utterances to convey past experience [3, 4].

Research AAC systems have attempted to support more natural conversation flow
by storing smaller conversational units so that the order of retrieval can vary. The
*Talk* system [5] demonstrated a retrieval system through which the user is supported
to navigate through a conversation. *Talk:About™* [4] included ways to introduce a
story, tell it at the pace required (with diversions) and give feedback to comments
from listeners; but again this tool was based on a library of fixed texts. Natural lan-
guage generation (NLG) has been used in more recent AAC projects. Dempster [6]
developed a system which uses information either entered by the user or retrieved
from sources such as online radio stations to create narratives. The *How was School
today...? (HwSt)* project [7] uses sensor data to provide information about a disabled
child's day at school. Time stamped data related to location (e.g. 11.05am: Hall),
people (e.g. 11.12am: Mrs Sound) and objects (e.g. 11.15am: Tambourine) were col-
lected using RFID<fnref n="1" /> sensors. Using additional information, e.g. the child's timetable,
this data is transformed into narrative utterances using data-to-text technology, e.g.
“In the morning, I was in the hall”, “Mrs Sound was there”, “I played the tambou-
rine”. The child is able to produce an evaluation that relates to the previous utterance by
selecting a positive (happy face) or negative (sad face) button after an utterance, e.g.
in relation to the last two utterances: “She is nice.” or “I didn’t enjoy it.”. HwSt pro-
vided a symbolic interface to support pre-literate individuals – many AAC users are in
danger of not acquiring literacy [1].

3 The Chronicles Project

The vision was to develop a system which would guide both literate and pre-literate
users in sharing lifelong narratives in an interactive conversation. The technical chal-
lenges for the proposed project were: i) to develop strategies to capture lifelong narra-
tives for storage in the system; ii) to provide an effective retrieval system so that users

<footnote n="1">¹ RFID: Radio Frequency ID</footnote>
can retrieve appropriate narratives; iii) to display the retrieved narrative utterances in a manner that makes sense to the user; and iv) to support automatic evaluation as in *HwSt* (“Jane was there” ⇨ “I like her”). An ethnographic study was conducted in a day centre for adults with physical and learning disabilities. This resulted in the establishment of story telling groups to support clients to develop narrative skills [8]. The design phase involved the co-design of an interactive story telling device with four nonspeaking participants [2].

# 4 The Chronicles System

The *Chronicles* system is designed to allow users to retrieve parts of a narrative non-sequentially to facilitate an interactive conversation instead of delivering a monologue. In addition, users can elaborate by selecting buttons to expand on the characters and places within the story and add emotional evaluations to utterances by choosing the appropriate buttons. Chronicles is implemented within the Tobii Communicator 4², an AAC software platform, and was installed on a DynaVox Vmax³, a rugged touchscreen tablet running Windows XP.

The interface (Fig. 1) follows a logical flow, starting with a story selection bar at the bottom of the screen, moving up into the utterances that make up that story in the middle of the screen, and at the top to the embellishments that can be made, allowing the user to elaborate on the story. Potential users of the Chronicles software might not have functional literacy; images are used in conjunction with a small amount of text to help users identify and understand each story, utterance, and embellishment.

<figure><img src="image_3.png" /><figcaption>Fig. 1. The final interface.</figcaption></figure>

The interface is divided into six sections:

1. Story Selection: Users can navigate through all of the stories they have stored in the system. When a story is selected, the *Chronicles* software retrieves all its utterances and populates the Utterance section of the screen with this data.

<footnote n="2">² http://www.tobii.com/en/assistive-technology/global/products/software/tobii-communicator/</footnote>
2. Utterances: The utterances are the parts that make up a story. Each utterance can be selected independently of the others, allowing a change of narrated order and repetition during conversation.

3. Embellishment Buttons: Allow the user to embellish on the story, providing them with the opportunity to talk about information that surrounds the narrative, such as where and when the story took place, but is not crucial to the story itself.

4. People Embellishment Buttons: If a person is mentioned in an utterance, a button with their picture links to utterances about that person.

5. Evaluation Buttons: The happy face button generates a positive comment and the sad face button generates a negative comment about the last utterance spoken. The evaluations are contextualised to the preceding utterance; if the utterance was "I had chicken for dinner", the positive evaluation button will generate a positive food experience evaluation, for example, "It tasted delicious!"

6. System Tools: The control buttons provide a variety of functions including selecting a similar story, editing the current story, switching to their existing AAC software (disabled during the evaluation period) and exiting the *Chronicles* system.

# 5 Evaluating Chronicles

Conversations with and without the Chronicles system were analysed to determine how the structure of narratives differs from those in using a traditional VOCA.

## 5.1 Participant

Gordon<fnref n="4" /> (31 years old) has cerebral palsy and uses a motorized wheelchair with a joystick. Gordon has no intelligible speech but uses some vocalisation and idiosyncratic gestures. He uses a Pathfinder<fnref n="5" /> device with Minspeak LLL 128; this system has a symbol keyboard interface – words are stored under symbol sequences. He uses direct selection to access keyboard sequences; this is labour intensive due to Gordon's physical difficulties. Retrieved words are spoken through a speech synthesiser. Gordon is a competent communicator, initiating conversations with communication partners. He has some difficulty shifting between topics and maintaining a prolonged conversation. He relies on producing single words or short sentences using his VOCA.

<footnote n="3">³ http://dynavoxtech.com/products/</footnote>

<footnote n="4">⁴ All names of participants are changed</footnote>

<footnote n="5">⁵ https://store.prentom.com/</footnote>
## 5.2 Methodology

The evaluation phase was primarily qualitative, with the main objective of assessing the performance and usability of the system in a care setting. Researchers spent an hour a week over a six week period in a local adult care setting, working with the participant. Narratives were elicited by encouraging the participant to talk about his past. This involved the researcher working with the participant and care worker using his existing VOCA (see [8] for details). The researcher then entered the data into *Chronicles*. He was then trained to use the *Chronicles* system and was given opportunities to practice any skills learned with both existing and novel partners. The participant was encouraged to use the system independently, but received support from a care worker when required. During each training session, the researcher set up the device to the *Chronicles* home screen and Gordon selected a story to tell or a topic for discussion. At the end of the evaluation phase, Gordon was asked to speak to two unfamiliar partners. All sessions were video recorded and subsequently transcribed using the following notation:

<table><tr><td>Talk</td><td>Natural speech</td><td>{ button } – Interface button selected</td></tr><tr><td>[ Talk – Overlapping speech</td><td>(( turns )) – Non-verbal interaction</td></tr><tr><td><u>Talk</u> – Whole word / phrase computer generated speech</td><td>???</td><td>– Inaudible speech</td></tr></table>

## 5.3 Equipment

During the training sessions with the *Chronicles* system, the participant had access to his own VOCA. This allowed him access to his full vocabulary for narrative elicitation purposes. During the evaluation sessions, the participant had access to either his VOCA or the *Chronicles* system, to facilitate a comparison of the systems.

## 5.4 Results and Analysis

Transcripts of conversations between Gordon and familiar and unfamiliar partners were analysed to identify whether there was any difference in the way in which conversational narratives were structured and how this impacted on the conversational flow. Using Labov's [9] definition of narrative, narratives were identified in the transcripts as a sequence of at least two clauses, which are temporally ordered, where any change in their sequence results in a change in the sequence of narrative events. Narrative sequences were then analysed in terms of having an abstract or summary, an orientation, evaluations, narrative clauses, a result and a coda (to mark the end of a narrative). Narratives also involve a planned disruption (i.e. what makes the event of interest), orientation to the listener (i.e. not a monologue), serialization of discourse markers (e.g. "and then..."), and context cues (providing background information). It was also important to analyse whether a narrative emerged during the conversation as an orally realised discourse unit [9]. Figures 2 and 3 show extracts of conversations, first using the participant's existing VOCA (Pathfinder) and then *Chronicles*.

Narratives stored within *Chronicles* were elicited during conversations with the participant using his VOCA (Pathfinder) with help from a care worker. Conversations
using the Pathfinder are characterised by the retrieval of individual words, e.g., ***No yesterday card David Hannah came.*** Each word is retrieved and then spoken by pressing a maximum of three keys sequentially. These sequences have been memorised by Gordon over time. Having retrieved the individual words, Gordon selects the “speak display” key to say the sentence as a whole (Fig. 2, line 1). Although Gordon is interacting with Kate, an unfamiliar partner, Gordon wanted Carol, his care worker, to participate. It soon became clear that Carol’s role was to act as “interpreter” as she worked with Gordon to understand that he wished to tell Kate that his cousin David (who he regards as a brother) forgot his birthday and didn’t even send a card. Much of the 14 lines of transcript are working out what Gordon is trying to communicate. However, Gordon does succeed in giving the vital details of his story. Carol tries to elaborate the story for Gordon in line 12, giving him the opportunity to evaluate his story nonverbally in line 13. The rest of the conversation (not shown) followed a similar pattern to reveal that Hannah is David’s wife. Having elicited the story from Gordon, it is not stored for retelling. In future, if Gordon wishes to retell his story, he will have to undergo the same process unless Carol and Kate are with him in which case he might defer to them to tell it for him. If Carol thinks the story is important, she might store all the parts of the story in the notebook facility of the device, but this would be retrieved as a monologue.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Transcript 1 – Gordon speaking to Carol (familiar) and Kate (unfamiliar) using Gordon’s existing VOCA (Pathfinder with Minspeak LLL 128).</figcaption></figure>

In comparison, the conversation with Richard in figure 3 follows a very different pattern. Unlike transcript 1, the conversation is not primarily concerned with unraveling what the aided conversation partner is trying to say. Instead, the conversation flow is more natural. Gordon spends less energy retrieving words. In transcript 1, he accesses 12 individual words and speaks three full utterances, necessitating 15 separate retrieval sequences (each one requiring at most three key sequences). Using *Chronicles*, he accesses 5 full utterances consisting 60 words using 5 retrieval button selections. The utterance in line 5 (figure 3) prompts a question and answer sequence in which Richard uncovers more detail.

The structure of the narrative reflects some of the characteristics of a typical conversational narrative. It begins with an abstract (line 1), has an orientation (at Christmas, when I was six), narrative clauses (several utterances), a result (line 18) and a
coda (line 27). Although Gordon did not use any evaluation buttons, Richard provides access to emotional evaluation in lines 17 and 27, similar to Carol and Kate in transcript 1 (lines 12 and 14). Both narratives involve a planned disruption (e.g. missing the birthday or wearing casts). Gordon responds to the listener in both conversations, e.g., line 3 of transcript 2, and some discourse markers are found in transcript 2 (e.g. “as well” in line 18). Much of transcript 1 attempts to elicit context cues (who is David?) while Gordon is able to provide background information in transcript 2.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Transcript 2 – Gordon speaking to Richard (unfamiliar) using CHRONICLES.</figcaption></figure>

## 6 Discussion

The purpose of this paper is to illustrate the potential of a narrative based system to facilitate a different type of conversation compared to that which is produced using a conventional word retrieval system. A single case study has been used to demonstrate the usage of *Chronicles*. Using a conventional device, Gordon's conversation is stilted with a focus on ascertaining the facts of the story. Using *Chronicles*, there is a more natural flow as the nonspeaking conversation partner is more independent of his contributions. Narratives are told using both systems, but different characteristics are more apparent when using *Chronicles*. Although Gordon did not use the evaluation buttons in this transcript, he has done so in other conversations. Analysis of the two transcripts, one with and one without the *Chronicles* system, shows differences in interactions; when using *Chronicles*, the user is better able to initiate topics, sustain topics, modify the narrative content and add emotional evaluation. The system allows
the user control when sharing personal narrative and demonstrates that a narrative-
based AAC system can facilitate more natural conversation. The analysis of the con-
versations illustrates how a narrative based AAC system can enhance the flow of an
aided conversation.

The results of this case study suggest several avenues for further research. The elic-
itation of past experience remains a challenge and work is underway to support input
of experience by users and carers by developing a framework through which past
experiences can be elicited, e.g., using an automated question and answer sequence to
elicit the parts of a narrative in terms of an abstract, an orientation, narrative clauses,
evaluations, a result and a coda. *Chronicles* is currently being ported to a tablet plat-
form and this will facilitate further case studies.

7 Acknowledgments

The authors would like to thank all the participants and staff at the day centre for
welcoming the researchers to share their lives for the duration of the project. This
project was funded by a Digital Economy Research Grant: EP/I031901/1. Project
web link – http://aac.computing.dundee.ac.uk/

References

1. Beukelman, D. R., Mirenda, P.: Augmentative and Alternative Communication: Manage-
ment of Severe Communication Disorders in Children and Adults. Baltimore, Paul H.
Brookes Publishing Co (2005)

2. Prior, S., Waller, A., Kroll, T., Black, R.: Use of an Agile Bridge in the Development of Assistive Technology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1579-1588 (2013)

3. Grove, N.: Using Storytelling to Support Children and Adults with Special Needs: Transforming lives through telling tales. Routledge, Oxford, UK (2012)

4. Waller, A.: Communication Access to Conversational Narrative. Topics in Language Disorders, 26(3), 221-239 (2006)

5. Todman, J., Alm, N.: Modelling conversational pragmatics in communication aids. Journal of Pragmatics, 35(4), 523–538 (2003)

6. Dempster, M., Alm, N., Reiter, E.: Automatic generation of conversational utterances and narrative for AAC: a prototype system, NAACL HLT 2010 Workshop on Speech and Language Processing for Assistive Technologies, 10-18, Log Angeles, USA, 1-6 June (2010)

7. Black, R., Waller, A., Turner, R., Reiter, E.: Supporting Personal Narrative for Children with Complex Communication Needs, ACM TOCHI, 19(2) (2012)

8. Prior, S., Waller, A., Black, R., Kroll, T.: Investigation of narrative use in the care envi-
ronment. Communication Matters, 41-42 (2012)

9. Labov, W.: Language in the Inner City. Philadelphia: University of Pennsylvania Press (1972)