# Graphical User Interface Redefinition Addressing Users' Diversity

José Luís Silva

<figure><img src="image_1.png" /></figure>

## To cite this version:

José Luís Silva. Graphical User Interface Redefinition Addressing Users' Diversity. Cristian Bogdan; Kati Kuusinen; Marta Kristín Lárusdóttir; Philippe Palanque; Marco Winckler. 7th International Conference on Human-Centred Software Engineering (HCSE), Sep 2018, Sophia Antipolis, France. Springer International Publishing, Lecture Notes in Computer Science, LNCS-11262, pp.319-326, 2019, Human-Centered Software Engineering. 10.1007/978-3-030-05909-5_20. [https://doi.org/10.1007/978-3-030-05909-5_20](https://doi.org/10.1007/978-3-030-05909-5_20). This work is licensed under a Creative Commons Attribution 4.0 International License.

HAL Id: hal-02270713

https://inria.hal.science/hal-02270713

Submitted on 26 Aug 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Graphical User Interface Redefinition Addressing Users’ Diversity

José Luís Silva¹, ² [0000-0002-1226-9002] and J. C. Silva³ [0000-0002-4575-0142]

¹ Madeira-ITI, Funchal, Portugal

² Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR-IUL, Lisboa, Portugal

³ Escola Superior de Tecnologia, DIGARC, Instituto Politécnico do
Cávado e do Ave, Barcelos, Portugal

jose.l.silva@m-iti.org, jcsilva@ipca.pt

**Abstract.** Improvements can still be made in the development of Interactive Computing Systems (ICSs) aiming to ease their use. This is particularly true when trying to address users’ diversity. Most ICSs do not adjust themselves to the user nor consider user’s particularities. However, some provide solutions to address better specificities of expert and novice users. Others adjust themselves based on user’s interaction history, but this does not always lead to improvements in use. An aspect that prevents to address users’ diversity broadly is the fact that most of existing ICSs do not provide source code access. This means that only owners can introduce improvements on them.

This paper proposes an approach (based on both affective computing and computer vision) to broadly improve design for diversity (without source code access) for both existing and to be developed ICSs. The results are twofold: i) example of an initial set of design guidelines; ii) opens the way to runtime Graphical User Interface (GUI) redefinition and adjustment based on both user’s features and emotions reducing therefore designers’ restrictions when addressing users’ diversity.

**Keywords:** Engineering Interactive Computing Systems; Graphical User Interface redefinition; Diversity Inclusion; Design Guidelines; Affective Computing.

## 1 Introduction

User interfaces (UIs) are a central aspect in any ICSs and critical for user's acceptance. According to the life cycle of a technology presented by Norman [1] there is a transition point where technology satisfies basic needs and customers are much more interested in efficiency, pleasure, and convenience. Therefore, UIs associated with a technology should be adjusted along time to meet user's expectations. Adjustment along time is also important while designing UIs because user's specificities are not always static, they change over time.
Even with the advances of user-centered approaches, design (e.g. universal design [2]) and usability (e.g. universal usability [3, 7]) issues persist [4]. According to Meiselwitz et al. [5] the challenges to achieve universal usability are associated to gap in users' knowledge and both technological and user diversity. Several approaches have been developed to take diversity into account when developing ICSs. Some consider that universal solutions should be considered while others ponder the use of specific designs targeted to specific user profiles as a way of embracing users' diversity. Both approaches have advantages and disadvantages. In one hand, designing for all users pose several challenges. One of the biggest challenges pointed out by Huh et al. [3] is finding the right balance between supporting all users and bringing enough profit to the designers. This is related to the fact that extreme users tend to be easily ignored due to their small number or simply because their existence is not always known. In the other hand, design for specific users or situations might deal better with extreme users but usually at the expense of compromised universality/globality.

This paper outlines a generic approach that allows the application of specific design guidelines to any individual or set of ICSs at runtime. The approach is based on the analysis of the graphical user interface (GUI) using both computer vision-based (Sikuli-based [6]) and affective computing-based approaches. Therefore, this allows the approach to be generic and without the need for ICS(s)' source code access. This enables developers and designers to improve the use of existing ICSs by empowering them to redefine existing GUIs and, consequently providing support for a wider diversity design inclusion. GUI adaptation without source code access poses several challenges as the information access and interaction should be made using computer vision-based techniques. In addition, the addressed GUI should be hidden or augmented to cope with the introduced modifications.

The article is structured as follows. Section 2 describes background concepts and presents some related research. Section 3 presents the proposed approach illustrated with an example in Section 4. Finally, discussion and conclusions are presented in Sections 5 and 6.

## 2 Related Work

Developing ICSs that address diversity of users pose several challenges. The term “universal design” describes the concept of designing all products to be ideally usable by everyone. Universal usability [7] was introduced with the goal of facilitating the use of interfaces. From those concepts several principles, guidelines, heuristics and standards were developed [8, 9, 10]. However, those approaches are sometimes not sufficiently complete to meet usability needs of specific users [4]. They might even be contradictory [13].

Those challenges lead to the proposal of a new set of solutions to support diversity mostly based on computer vision. The ISI (Interactive Systems Integration) tool [14] enables the integration of several GUIs of different ICSs into one new GUI adapted to user's characteristics. However, features such as collecting the state of the original GUIs are still primitive. The work of Dixon et al. enables the identification of some
GUI widgets [15, 16]. From this identification it is possible to build GUI pixel-based enhancements [15]. One example presented by the authors is the automatic translation of a GUI using Prefab, a tool using computer vision algorithms. The approach provides GUI pixel-based enhancements (to a set of GUI widgets) but not its redefinition. We argue that supporting attentive real-time GUI redefinition is of major importance to foster design for diversity. Other works [17, 18] like SUPPLE [19] enable automatic GUI generation adapted to a person's specificities but it does not enable GUI redefinition of existing ones.

Several solutions from the computer vision field provide basis for GUI's widgets identification (e.g. OpenCV<fnref n="1" />, CVIPtools²). For example, neural networks YOLO [20] is a real-time object detection solution that can detect over 9000 object categories from an image or video. Unfortunately, it was not applied to GUI widgets. We believe that this approach can be successfully applied to support GUI redefinition, but we are unaware of any work with this purpose or any annotated GUI widgets database essential for this purpose.

Emotion has gain importance in Human-Computer Interaction. Affective Computing [21] in particular addresses several challenges concerning with computers and emotions (e.g. ability to recognize and express emotions). The work of Tao et al. [22] and Poria et al. [23] provide an adequate review. Works in the area of affective interfaces consider emotion in the design (e.g. [24]). For example, the work of Mori et al. [25] reports results that aim to improve the understanding about what design techniques are more important to stimulate an emotion on the user. Alexander et al. [26] outlined the plan for an interface that adapt like humans to the non-verbal behavior of users. The idea has similarities with our approach however our focus is on GUI redefinition based on user's emotions but also based on other user's features (e.g. disease, experience).

## 3 GUI Redefinition and Design Guidelines

Prior work done by Gaganpreet et al. [27] developed an emotional state estimator that provides input for runtime GUI redefinition in the context of life critical robot teleoperation. This work is a complement to the Prefab's approach because users' emotions can be considered. However, the GUI redefinition is done manually case by case and without any guidelines.

Complementing those works enables a generic GUI redefinition for all approach fed by the user emotional state. This is beneficial because it enables designers and developers to keep existing design and source code but without preventing them to redefine the GUI to the diversity at runtime. Those advances might be further improved if user's specificities are considered. In addition, the development of a Design Guideline Provider (see Figure 1) suggesting automatic GUI redefinitions based on both emotional user's states and specificities represent also an improvement.

<footnote n="1">¹ opencv.org (last accessed June 4, 2018).
² cviptools.ece.siue.edu (last accessed June 4, 2018).</footnote>
Figure 1 presents the architecture of our GUI redefinition approach. It is composed by three main components that enable the proposed redefinition. They are:

1. Emotional State Estimator that identifies user's emotional states from their physical monitorization (e.g. via affectiva<fnref n="3" /> and imotions<fnref n="4" />);

2. User's Specificities Identifier that identifies the user's profile and personality (Myers-Briggs Type Indicator<fnref n="5" />) from the answers provided to the questionnaire and from results of the test task performed;

3. Design Guidelines Provider that suggests design guidelines for the GUI redefinition based on the identified emotional states and user's specificities.

The output of the Design Guidelines Provider supports an automatic GUI redefinition at runtime. The old GUI is hidden running in a virtual machine and only the new GUI is presented to the user. We follow the approach made by Silva et al. [14] to enable a transparent GUI redefinition for the user. Silva et al. developed ISI, a tool that enables the creation of a new UI abstraction layer integrating different ICSs without accessing their source code. The proposed integration aims to improve end user interaction. The tool uses enriched ConcurTaskTree models and selected scenarios to generate Sikuli scripts. Each script is then associated to a widget of the new GUI. The interaction with a widget of the new GUI triggers the execution of the associated Sikuli script that will perform the task on existing ICSs.

<figure><img src="image_3.png" /><figcaption>Fig. 1. GUI redefinition approach.</figcaption></figure>

<footnote n="3">³ affectiva.com (last accessed June 4, 2018).</footnote>

<footnote n="4">⁴ imotions.com (last accessed June 4, 2018).</footnote>

<footnote n="5">⁵ psycnet.apa.org/record/2013-29682-000 (last accessed June 4, 2018).</footnote>
This work enables an innovative approach that address diversity more effectively. For this an example of an initial set of guidelines (mapping between user's specificities / emotional state and redefinition rules) for the Design Guidelines provider are presented below.

**Design Guidelines for some detected user's specificities:**

1. Novice: step by step with tutorial;

2. Expert: automation;

3. Parkinson: selection via gaze, increase size;

4. Blind: translation rules;

5. Deaf: augmented captions.

**Design Guidelines for some detected user's emotional states:**

1. Frustrated: more effective interaction;

2. Confused: clarification, more feedback;

3. Stressed: use additional communication channels (e.g. scent, relaxing music) to appease the user;

4. Overload: split information into different communication channels (e.g. visual, audio, haptic).

This is only an example of initial set of guidelines. Machine learning algorithms will be developed to identify better guidelines based on the combination of the set of inputs detected (user's specificities and emotional states) and users' reaction. The resulting guidelines will be used as a basis for better automatic GUI redefinition.

We aim at enriching GUIs with emotional intelligence. This means that in the same way a person usually adapts to its interlocutor while interacting with him/her. Our approach enables a GUI to automatically adapt at runtime to the user that is interacting with it.

## 4 Example

This section as the purpose of illustrating the value of the approach with a concrete example where the opening for user diversity is made clearer.

Consider an application running on a tablet while its user is travelling by train. Due to the vibrations of the train the user becomes frustrated because he can't always hit the desired widget at the first attempt. In addition, the analysis of the interaction made revealed that the user has motor difficulties. Therefore, the input provided to our Design Guidelines provider (frustrated mental state and motor difficulties) lead to design guidelines suggestions such as:

* increase the size of icons;

* associate wrong clicks to the closer widget;

* enable gaze interaction.

Those guidelines are then automatically applied at runtime for the GUI redefinition. The GUI is updated when user emotional state changes are detected.
# 5 Discussion

Most contributions tend to develop models, applications or technologies to improve the design and development of new and better ICSs. This work aims to facilitate the improvement of existing ones without accessing their source code. One can argue that (old) existing ICSs tend to disappear being replaced by new versions. The fact is that several systems or old versions are still in use today. Some of them will eventually be updated in the future but the remaining ones will not. This might be explained by two reasons: i) absence of updated versions; ii) the user does not update it.

Several ICSs do not enable personalization nor were designed to consider diversity. Furthermore, as integration is also considered, this approach enables the application of specific or universal solutions to the set of GUIs to be integrated. Ultimately, this can lead to a new way of designing and developing ICSs where they are used, as a whole, in task and user centered design approaches. For instance, several pieces of existing software with different purposes can be merged together into a new GUI where the goal is not to expose the functionalities of the system but rather use them to empower the user to accomplish easier tasks execution.

The design of GUIs to be run on top of existing ICSs raise implementation challenges but we believe that the advantages are strong:

1. GUIs enhanced with emotional intelligence adjusting themselves to the user who is interacting with it;

2. Enable design for diversity to be automatically applied at runtime to new and existing GUIs.

Challenges of using computer vision-based techniques such as disambiguation in the identification of widgets can be reduced with anchors (UiPath<fnref n="6" /> software follow this method). Theme variations (e.g. color, text font, size) might introduce difficulties while running the Sikuli scripts. These can be reduced by introducing transparencies in the figures provided to the scripts.

It is important to note that, although some parts are already implemented, the presented approach is an ongoing work. The solution and guidelines must be evaluated with user studies. For instance, GUI runtime modifications in one direction can be fast when detecting user confusion but should be slow when the user emotional state is going back to user “normal” state. The identification of an adequate delay to be used in this case is an example of the importance that the user studies will have.

# 6 Conclusions

This paper presents an approach that aims at enabling developers and designers to change any existing GUI at runtime based on both user's specificities and emotional state. This will ultimately enable designing for diversity to be broadly applied. In

<footnote n="6">⁶ uipath.com (last accessed July 4, 2018).</footnote>
addition, an example of a set of design guidelines based in this innovative approach is
presented.

## Acknowledgments

This work was supported from Fundação para a Ciência e a Tecnologia (FCT, Portugal), through project UID/EEA/50009/2013 and by ISTAR-IUL through project UID/MULTI/0446/2013.

## References

1. Norman, D. A. (1998). The invisible computer: Why Good Products Can Fail, the Personal Computer Is So Complex, and Information Appliances Are the Solution. Cambridge, MA: MIT Press.

2. Law, Chris M., et al. (2008). "A systematic examination of universal design resources: part 1, heuristic evaluation." Universal Access in the Information Society 7.1-2, 31-54.

3. Jina Huh and Mark Steven Ackerman. (2009). Designing for all users: including the odd users. In Extended Abstracts on Human Factors in Computing Systems. 2449-2458. ACM.

4. Ruzic L., Sanfod J.A. (2017) Universal Design Mobile Interface Guidelines (UDMIG) for an Aging Population. In: Marston H., Freeman S., Musselwhite C. (eds) Mobile e-Health. Human-Computer Interaction Series. Springer, Cham

5. Meiselwitz, G., Wentz, B., Lazar, J. (2009). Universal usability: Past, present, and future. Foundations and Trends in Human-Computer Interaction. Now Publishers Inc.

6. Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. (2009). Sikuli: using GUI screenshots for search and automation. Symposium on User interface software and technology. ACM, 183-192.

7. Ben Shneiderman. (2000). Universal usability. Commun. ACM 43, 5, 84-91.

8. Shneiderman, Ben. (2009). "Shneiderman's eight golden rules of interface design".

9. FDIS, ISO. "9241-110: 2006. Ergonomics of human system interaction-Part 110: Dialogue principles." International Organization for Standardization (ISO). Switzerland.

10. ISO. Ergonomics of Human-system Interaction: Principles and requirements for physical input devices (ISO 9241--400:2007, IDT). International Organisation for Standardisation, 2007.

11. Vanderheiden, Gregg C., and J. Bern Jordan. "Design for people with functional limitations." Handbook of Human Factors and Ergonomics, Fourth Edition (2012): 1407-1441.

12. C., Douglas E., S. Zavotka. Aging, disability, and frailty: implications for universal design. Journal of physiological anthropology 25.1 (2006): 113-118.

13. R. Pak, and A. McLaughlin, Designing displays for older adults: CRC Press, 2010.

14. Silva, J. L., Ornelas, J. D., & Silva, J. C. (2016). Make it ISI: interactive systems integration tool. In Proc. of the Symposium on Engineering Interactive Computing Systems (pp. 245-250). ACM.

15. Morgan Dixon, Daniel Leventhal, and James Fogarty. (2011). Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 969.

16. Morgan Dixon, A. Conrad Nied, and James Fogarty. (2014). Prefab Layers and Prefab Annotations: Extensible Pixel-Based Interpretation of Graphical Interfaces. UIST '14, 221-230.
17. Krzysztof Z. Gajos, Daniel S. Weld, and Jacob O. Wobbrock. (2010). Automatically generating personalized user interfaces with Supple. Artif. Intell. 174, 12-13, 910-950.

18. Krzysztof Z. Gajos, Jacob O. Wobbrock, and Daniel S. Weld. (2008). Improving the performance of motor-impaired users with automatically-generated, ability-based interfaces. In Proceedings of Conference on Human Factors in Computing Systems. ACM, USA, 1257-1266.

19. Krzysztof Gajos and Daniel S. Weld. (2004). SUPPLE: automatically generating user interfaces. In Proceedings of the 9th international conference on Intelligent user interfaces. ACM, 93-100.

20. Redmon, Joseph, and Ali Farhadi. (2016). "YOLO9000: better, faster, stronger." arXiv preprint arXiv:1612.08242.

21. Picard, Rosalind Wright. (1995) Affective computing. The MIT Press.

22. Tao, J., & Tan, T. (2005). Affective computing: A review. In International Conference on Affective computing and intelligent interaction (pp. 981-995). Springer, Berlin, Heidelberg.

23. Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98-125.

24. Cyr, D.: Emotion and web site design. In: Soegaard, M., Dam, R.F. (eds.) The Encyclopedia of Human Computer Interaction, 2nd edn., (Chapter 40), Interaction Design Foundation.

25. Mori, G., Paternò, F., & Furci, F. (2015). Design Criteria for Stimulating Emotions in Web Applications. In Human-Computer Interaction (pp. 165-182). Springer, Cham.

26. Alexander, S., & Sarrafzadeh, A. (2004). Interfaces that adapt like humans. In Asia-Pacific conference on computer human interaction (pp. 641-645). Springer, Berlin, Heidelberg.

27. G. Singh, S. Badia, R. Ventura and J. L. Silva. (2018) Physiologically Attentive User Interface for Robot Teleoperation - Real Time Emotional State Estimation and Interface Modification using Physiology, Facial Expressions and Eye Movements. In Proc. of the International Joint Conference on Biomedical Engineering Systems and Technologies. ISBN 978-989-758-279-0, pages 294-302.