# Human-Computer Interface Design of a Communication Aid Software System for People with Parkinson's Disease

Yohan Guerrier¹, Sophie Lepreux¹, Christophe Kolski¹, Véronique Delcroix¹, and Káthia Marçal de Oliveira¹

¹Univ. Polytechnique Hauts-de-France, CNRS, UMR 8201 LAMIH, F-59313 Valenciennes, France

**Abstract**

Easy to use, intuitive and adaptive; with these three requirements defined by a neurologist we started the development of a communication aid software system for people with Parkinson's disease, named *ParkinsonCom*. We understood that the Human-Computer Interaction (HCI) would be a key issue. Following HCI design principles, we decided to design the user interfaces, make mock-ups and validate them with users before implementation. This paper presents how this process was accomplished in a totally distant way (by internet) due to the confinement, and the results obtained.

**Keywords**

Communication, Assistive technology, disability, Parkinson

## 1. Introduction

Parkinson's disease is the second most common neurodegenerative disease in the world. The quality of life and social participation of people with Parkinson's disease (PD) are severely affected, not only by motor symptoms, but especially by non-motor symptoms such as communication disorders that impair speech intelligibility, the ability to express emotional states and, consequently, social relationships. In order to improve this situation, we have been working on the development of a communication aid software system for people with PD in the ParkinsonCom project [1].

Using a user-centered design approach [2], we developed a first prototype. To that end, a process of design, mockup and validation of the user interface was performed. Our main challenge for this was to accomplish all these issues remotely via internet meetings due to COVID's health crisis. Following the standard NF EN ISO 9241-210 (2011) [3] concerning the user-centered process, a survey was carried out to identify the main difficulties encountered by people having PD. Next, we made interviews with experts (11 persons with PD, 7 spouses of persons with PD and 3 neurologists). After extracting the essential needs of people with PD, we defined the priorities about functionalities to be implemented in our prototype:

* Make a dialog - made for PD patients to communicate with people, they can type a dialogue using the keyboard and the pictograms suggested while typing in text area. Those dialogs can be reused later by accessing the appropriate menu.

* Make a dialog based on suggested ones from library: we propose some ready dialogs that can be used, modified or completed.

* Use a dialog - saved dialogs or proposed ones can be modified or read by voice synthesis.

* Short request - Request for help to a specific person by a call, especially in off mode (a period of blockage or discomfort) [4]. Besides, users can access this screen to entertain themselves by listening to music via an audio player, or listen to funny stories.
With those defined functionalities, we had to design the user interfaces before the final implementation. Our main difficulty was how to define user interfaces without meeting the users. With the period of confinement, any face-to-face interaction was prohibited, especially with users with a risk profile (elderly people with some illness). How to guarantee simple and intuitive interfaces? We decided therefore to set up a workshop with psychologists and psycho-educators who were partners in the project to draw these user interfaces; then, to make mockups of them; and finally, to present and validate them with final users (people with Parkinson's disease) remotely.

This article reports this experience of defining user interfaces for the ParkinsonCom software. Section 2 presents a brief background, before describing the workshop performed for brainstorming about the user interface design (section 3) and the mockups defined from the results of the workshop (section 4). Section 5 presents the results of the validation of the mockups with people with Parkinson. Finally, section 6 shows our next steps.

## 2. Background

Convinced that we should use a user-centered approach, we asked ourselves about which techniques and procedures to apply and which technologies to use. To answer this question, we have performed a rapid review [5] following the guidelines defined by [6] with the objective of analyzing the studies performed for communication tools or in the field of Parkinson Disease.

Applying a specific search string ("user-centered" OR "human-centered" OR "user-centered" OR "human-centered") AND ("communication" OR "conversation" OR "ICT" OR "parkinson")) in SCOPUS database, we obtained 1576 documents, that after the elimination of duplicates, resulted in 1,548 documents to analyze. With a reading procedure searching only articles for our objective we obtained 67 final articles.

Figure 1 shows the different approaches of user-centered design to capture the needs of future users, as well as to proceed with the design and evaluation. The three most used methods are interviews, prototype testing and workshops.

<figure><img src="image_1.png" /><figcaption>Figure 1: User-centered design approaches</figcaption></figure>

Of the 29 articles mentioning the use of interviews, we found that they were used with Parkinson's patients, people with Autism Spectrum Disorder (ASD), people with mental disabilities, elderly people, and people with speech impairments in general. The majority of the articles concerned applications related to communication support (e.g. [7], [8]). Of the 17 articles dedicated to prototype testing, the approach most often used is to create a prototype to identify needs, then test and improve it. We also identified 17 articles that cite the use of workshops to identify needs and requirements. Experts and future users usually participate in those workshops to discuss an application under development.

The rapid review showed also that the most used technologies are smartphones (in 32 of the selected articles), tablets (20) and computers (17). If we consider only the works for people with PD, the most used technological support is the tablet (10 of 13). These technologies are primarily used through the touch interaction mode in a large majority of cases (37). In some of these works the alternative and augmented communication (AAC) is presented as an effective interaction mode. For instance, Ilyas [9] developed a communication tool (pad communication) for people with disabilities, particularly having speech inhibition to communicate. Grigis and Lazzari [10] showed the importance of the AAC on tablets
in the life of persons with severe communication disabilities. Finally, the research project presented in [11] aimed to understand the needs for the design of a physician-patient communication visualization system, as well as to develop a prototype.

## 3. Workshops with Project Team

Based on the results of the rapid review and following the advices of the neurologists, we decided to develop the tool for tablets and use AAC. In order to establish the user interface layout, two design workshops were organized with psychologists, psycho-educators, HCI specialists and an expert in communication tools for people with disabilities. These allowed a number of suggestions to be made regarding the functioning and presentation of the system.

The first workshop focused on the main features such as:

1. Starting the software;

2. Preparing a dialogue from words/pictograms;

3. Preparing sentences or word lists from existing sentences;

4. Make a request when in off mode.

This allowed for discussions about certain choices and items position, which also demonstrated the need to make available choices that contribute to the personalization of the support system.

Various interface elements were discussed and drawings were made (See examples in **Figure 2**), for instance:

* the decision of whether the application should be started in the user's ON or OFF mode, or whether we should leave the choice to each;

* to support "prepare a dialogue" functionality, a draw was proposed focusing on the whole sentences recorded (by default or by the person). Then the preparation of a dialogue could consist in proposing sequences of words by following tree structures. The addition of one's own words could be done by manual additions, downloads from the internet or by taking personal photographs.

* One of the proposed draws shows how the user interface could look like to prepare the dialogue from sentences, words and pictograms. This mockup shows the importance of providing recording capabilities at any time, and that words can be added at any time. We have noted the importance of allowing the users to choose the voice of the speech synthesizer if they cannot record their own voice.

* In another draw, everyday sentences are pre-configured so that they can be easily adapted to the context. If the user chooses "Hello X, how are you?", the interface suggests replacing X by names, nicknames of people around him or her. It also suggests a follow-up so that the conversation is dynamic and not just a closed response.

<figure><img src="image_2.png" /><figcaption>**Figure 2:** Examples of three very different technics for interface drawings used during the workshop</figcaption></figure>

The second workshop was positioned around three main ideas:

1. We have lists of words, sentences, pictograms already recorded: how to use them?

2. We can think in the form of tasks that can be carried out by people.

3. Propose models using playful mechanisms.

These discussions sought to identify how to motivate the user with Parkinson's disease to use the tool to prepare dialogues. The idea would be to do it in a playful way, as if it was a game. The user, as
player, gives a beginning of a sentence, ending with an important word that is missing, the system tries to propose solutions (continuation of the sentence), and the player chooses a solution. If the word under the end of the sentence is not in the proposals, you can go to a help, not necessarily with words, but which can contain pictograms. It is always necessary to be able to go to a category directory to find something. We must also be able to complete the content of this directory, if a word is missing. Scoring and medal mechanisms common in online games can be used for users who make more dialogues, enrich the general dialogues base, and so on.

## 4. Mockups Design

Based on the discussions and results of the workshop, we created mockups for the "Make a Dialogue" functionality by presenting different options for Human-Computer Interaction (See for example, **Figure 3** and **Figure 4**).

<figure><img src="image_3.png" /><figcaption>(a) Landscape user interface with horizontally aligned buttons</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(b) Portrait user interface with vertically aligned buttons</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(a) Only word</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Only pictogram</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(c) Both pictogram + word</figcaption></figure>

<caption>**Figure 4:** Different options to support “Make a dialogue” functionality</caption>

## 5. Evaluation with people with Parkinson's disease
We performed mockup evaluations with 16 volunteer persons with Parkinson's disease (8 French and 8 Belgian). Each interview lasted on average one hour. The interviews had two objectives. The first objective was to present the project and its progress. The second objective was to identify the preferences on some options for Human-Computer Interaction of the users concerning:

* button placement (horizontally aligned at the top or vertically on the side);

* the positioning of the interfaces on the tablet (landscape or portrait);

* the way suggestions are presented to complete the text (text only, pictogram only or pictogram and text);

* the type of keyboard (classic or swiftkey which presents word suggestions);

* the choice of a symbol for the identification that they are in off mode and therefore, the choice to make a simple request (thumb down, smile, sad or text).

In addition, we sought to identify whether the chosen features would meet expectations and suggestions for prototype development. **Figure 5** shows the percentages of the different options on the presented human-machine interaction elements. We can see that the chosen elements are:

(a) the screen presented in landscape (75%);

(b) the buttons aligned horizontally at the top (63%);

(c) the swiftkey keyboard (81%);

(d) the use of pictograms followed by word(s) for the proposals to complete the text (50%);

(e) the thumb for the indication of off mode and therefore to allow to "make a request" (44%).

<figure><img src="image_8.png" /><figcaption>**Figure 5:** Options on Human-Computer interaction elements</figcaption></figure>

# 6. Conclusion

This paper reports the user-centered design of the user interfaces of a communication aid software system for people with Parkinson's disease. After this design, a first prototype of the system was developed and it was evaluated by final users. A new version of the system is under development to integrate gamification elements and personalization options.

# 7. Acknowledgements
ParkinsonCom project is developed with the support of the European Regional Development Fund (Interreg V France-Wallonie-Vlaanderen) and the Agency for quality life AVIQ (l'Agence pour une Vie de Qualité) from Wallonia, for which the authors are deeply grateful.

## 8. References

[1] K.M. Oliveira, E. Batselé, S. Lepreux, E. Buchet, C. Kolski, M. Boutiflat, V. Delcroix, H. Geurts, K. Apedo, L. Dehon, H. Ezzedine, Y. Guerrier, M. Haelewyck, N. Jura, P. Pudlo, Y. Rekik. ParkinsonCom Project: Towards a Software Communication Tool for People with Parkinson's Disease. In: International Conference on Human-Computer Interaction Springer, 2021, pp. 418-428.

[2] S. Lepreux, K. Apedo, K. Oliveira. Vers une conception centrée sur l'utilisateur ayant un profil évolutif : Une étude de cas avec des personnes atteintes de la maladie de Parkinson. In: Proceedings of the 31st Conference on l'Interaction Homme-Machine: Adjunct, ACM, 2021.

[3] NF EN ISO 9241-210 (2011-01-01) Ergonomics of human-system interaction - Part 210 : human-centred design for interactive systems (2011).

[4] J. Whitfield, A. Reif, and A. Goberman. Voicing contrast of stop consonant production in the speech of individuals with Parkinson disease ON and OFF dopaminergic medication. Clinical linguistics and phonetics 32, 7, 2018, pp. 587-59.

[5] Y. Guerrier, K. Oliveira, C. Kolski, A. Lepreux, K. Apedo, V. Delcroix, H. Ezzedine. Une étude systématique pour la conception d'un système d'aide à la communication pour les personnes atteintes de la Maladie de Parkinson. In: Actes du 39ème Congrès INFORSID, INFormatique des Organisations et Systèmes d'Information et de Décision, 2021, pp. 71-86

[6] A. C. Tricco, E. V. Langlois, S. E. Straus et al. Rapid reviews to strengthen health policy and systems: a practical guide. World Health Organization, Geneva, 2017.

[7] S. E. Martins, P. Maldonado. Cubo: Communication System for Children with Autism Spectrum Disorders. In: International Conference on Applied Human Factors and Ergonomics (pp. 353-365). Springer, 2019.

[8] K. Wołk, A. Wołk, W. Glinkowski. A cross-lingual mobile medical communication system prototype for foreigners and subjects with speech, hearing, and mental disabilities based on pictograms. Computational and mathematical methods in medicine, 2017.

[9] C.M.A. Ilyas, K. Rodil, M. Rehm. Developing a User-Centred Communication Pad for Cognitive and Physical Impaired People. In: Brooks, A., Brooks, E. (eds) Interactivity, Game Creation, Design, Learning, and Innovation. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 328. Springer, 2020.

[10] D. Grigis, M. Lazzari. Augmentative and alternative communication on tablet to help persons with severe disabilities. In: Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI (CHItaly '13). Association for Computing Machinery, 2013, pp. 1-4.

[11] F. Lin, C. Lin, Y.J. Hsueh, C. Lee, C. Hsieh. Graphical Tools for Doctor-Patient Communication: An App Prototype Design in Children's Pain Management. HCI, 2016.