# On-the-Fly Usability Evaluation of Mobile Adaptive UIs
Through Instant User Feedback

Enes Yigitbas, Ivan Jovanovikj, Klementina Josifovska, Stefan Sauer, Gregor Engels

<figure><img src="image_1.png" /></figure>

## To cite this version:

Enes Yigitbas, Ivan Jovanovikj, Klementina Josifovska, Stefan Sauer, Gregor Engels. On-the-Fly Usability Evaluation of Mobile Adaptive UIs Through Instant User Feedback. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.563-567, 10.1007/978-3-030-29390-1_38. hal-02877645

HAL Id: hal-02877645

https://inria.hal.science/hal-02877645

Submitted on 22 Jun 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# On-the-fly Usability Evaluation of Mobile Adaptive UIs through Instant User Feedback

Enes Yigitbas, Ivan Jovanovikj, Klementina Josifovska, Stefan Sauer, and Gregor Engels

Paderborn University
Fürstenallee 11, 33102 Paderborn, Germany
firstname.lastname@upb.de

**Abstract.** Adaptive User Interfaces (AUIs) have been promoted as a solution for context variability due to their ability to automatically adapt to the context-of-use at runtime. For the acceptance of AUIs, usability plays a crucial role. Classical usability evaluation methods like usability tests, interviews or cognitive walkthroughs are not sufficient for evaluating end-user satisfaction of AUIs at runtime. The main reason is that the UI and context-of-use are dynamically changing and acceptance of UI adaptation features has to be evaluated in the moment and context-of-use when the adaptations occur. To address this challenge, we present an on-the-fly usability evaluation solution that combines continuous context monitoring together with collection of instant user feedback to assess end-user satisfaction of UI adaptation features. We demonstrate the usefulness and benefit of our solution based on a mobile Android mail app, which served as basis for conducting a usability study.

**Keywords:** Adaptive User Interface · Usability Evaluation

## 1 Introduction

Adaptive UIs (AUIs) have been promoted as a solution for plasticity [3] due to their ability to automatically adapt to the context-of-use at runtime. With regard to AUIs, usability evaluation of UI adaptations is still a challenging task, especially in the ubiquitous domain of mobile UI platforms, where dynamically changing context-of-use situations are usual. In the past, classical usability evaluation methods like usability tests, interviews or cognitive walkthroughs were applied to evaluate the usability of AUIs [2]. However, these methods are not sufficient for a proper evaluation of dynamically changing UI adaptation features at runtime. The reason is that these methods mostly focus on a *posteriori* analysis techniques. However, the acceptance of each UI adaptation feature should be evaluated at the very moment and in context-of-use when the adaptation is triggered at runtime. To address this issue, we present an on-the-fly (OTF) usability evaluation solution that integrates UI adaptation features and a user feedback mechanism into a mobile app. The developed solution enables us to continuously track various context information data and collect user feedback, e.g. whether the users like or dislike the triggered UI adaptations.
The rest of the paper is structured as follows: In Section 2, we present the general solution idea for OTF usability evaluation of UI adaptations. Section 3 provides an overview of the concrete implementation of our adaptive mail app, which was used as a basis setup to perform an empirical experiment for evaluating AUIs. In Section 4, we describe the conducted usability study. Finally, Section 5 concludes our work with an outlook on future work.

## 2 Solution Idea

Our OTF usability evaluation solution targets rule-based UI adaptation approaches as introduced in our previous work [5]. We extended our existing rule-based UI adaptation approach with capabilities to support continuous context monitoring and collect context-driven instant user feedback.

Figure 1 illustrates the main idea of our OTF usability evaluation solution for UI adaptations. Based on IBM's MAPE-K loop [1], we have two main components *Adaptation Manager* and *Managed Element*. The *Adaptation Manager* is responsible for monitoring and adapting the *Managed Element*, which in our case is the user interface of a mobile platform. For supporting OTF usability evaluation of UI adaptation features, the existing monitoring and adaptation loop described in our previous work [5] needs to be extended. To this end, we

integrated an instant user feedback mechanism into the context monitoring and UI adaptation loop. The feedback mechanism allows users to explicitly rate the triggered UI adaptations. Users can give positive or negative feedback, but they can also ignore the feedback mechanism and concentrate on their main application task. As Figure 1 shows, there is a *Knowledge Base* which is responsible for storing all context information, all triggered UI adaptation rules, and the corresponding instant user feedback. Based on the stored information, it is possible to analyze the acceptance of UI adaptations based on the current context of the user and the user's feedback.

<figure><img src="image_3.png" /><figcaption>Fig. 1. On-the-fly usability evaluation of UI adaptation features</figcaption></figure>

## 3 Implementation

We implemented our on-the-fly usability evaluation solution based on an Android mail app using React Native<fnref n="1" />. We decided to use a mail app as email clients are

<footnote n="1">¹ https://facebook.github.io/react-native/</footnote>
one of the most frequently used apps on mobile platforms in various context-of-use situations. The implemented email app is based on Gmail and provides full email services. Beside the core mail application logic, inspired by [4], we implemented 28 UI adaptation features that cover different adaptation strategies like task-feature set-, layout-, navigation-, and modality-adaptation.

Figure 2 depicts three screenshots of the mail app showing the effect of exemplary UI adaptation features. The first screenshot in Figure 2 shows a minimalistic UI where an icon-based representation is shown. In contrast to that, the second screenshot shows a grid layout that could be useful for novice users. The third screenshot shows an example for UI modality adaptation where a switch from

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** UI Adaptations: minimalistic UI, "large" UI, and vocal UI (left to right)</figcaption></figure>

the graphical UI (GUI) to the vocal UI (VUI) was made and mails are read out in the language of the mail. To realize the described UI adaptations, we integrated the Nools<fnref n="2" /> rule engine into the mobile email app. It is fed with context information from various integrated sensors and cameras of the mobile platform. For example, we use different sensors like ambient light, accelerometer, gyroscope, battery or network sensor. Furthermore, we use timer, device information or existing recognition services such as *Amazon Rekognition*<fnref n="3" /> to derive the user's emotion as well as estimated age or gender based on camera photos.

For realization of the feedback mechanism, we integrated a feedback prompt into the adaptive email app. The left screenshot in Figure 3 shows how the feedback prompt was placed in the mail app. On the top of the screen the feedback prompt is shown whenever context changes were detected that lead to UI adaptations. The triggered UI adaptations are explained in the feedback prompt and the user is able to provide feedback by clicking the positive or negative smiley indicating whether the user liked the UI adaptation or not. In some

cases, for example, when the user is in a bad mood and the app detects this via camera, a feedback prompt in the form of a text field appears (see right screenshot in Figure 3) that allows the users to provide more detailed feedback.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Feedback prompts</figcaption></figure>

<footnote n="2">² http://noolsjs.com/</footnote>

<footnote n="3">³ https://aws.amazon.com/de/rekognition/</footnote>
# 4 Usability Study

During the usability experiment, the developed app was used for one week by 23 participants. All users were made aware of the fact that their interaction with the app would be closely monitored (e.g. using facial recognition). During the usability experiment, various data about the users and their usage context, while feedback was given, have been collected to evaluate the usability of the UI adaptation features in detail. In the following, we will shortly describe some of the collected data to show the potential of our usability evaluation solution. During the conduction of the experiment, there were 104404 detected context changes from all devices. Of these, only 37465 triggered an adaptation by the rule engine. However, users gave feedback on the adaptation rules in only 663 cases. Every time an adaptation rule received feedback, the previous context additionally to the current context was saved. In total, the users gave positive feedback in 616 cases and negative feedback in 47 cases. With about 93% of the feedback provided by users being positive this means that most of the user interface adaptations were liked by the users.

# 5 Conclusion and Outlook

This paper presents a novel solution and demo for evaluating the usability of UI adaptations for mobile platforms with a special focus on end-user satisfaction. We introduce an on-the-fly usability evaluation solution that incorporates instant user feedback and the corresponding context-of-use. In this way, our solution enables us to continuously track various context information about the user, platform, and environmental characteristics and user feedback, e.g. if the users accept the triggered UI adaptations or not. For future work, we plan to extend our usability evaluation approach to cover further usability aspects like efficiency and effectiveness. In this regard, further long-term usability studies with a larger user group is planned.

# References

1. An architectural blueprint for autonomic computing. Tech. rep., IBM (Jun 2005)

2. van Velsen et al., L.: User-centered evaluation of adaptive and adaptable systems: a literature review (2008)

3. Calvary, G., Coutaz, J., Thevenin, D.: A Unifying Reference Framework for the Development of Plastic User Interfaces (2001)

4. Paternò, F.: User interface design adaptation. In: The Encyclopedia of Human-Computer Interaction, 2nd Ed., chap. 39. Aarhus, Denmark (2013)

5. Yigitbas, E., Stahl, H., Sauer, S., Engels, G.: Self-adaptive UIs: Integrated model-driven development of UIs and their adaptations (2017)