wide number of available devices and implementation languages. Despite such
potential benefits, their adoption has mainly been limited to professional designers,
but new solutions have recently been emerging that are able to extend such
approaches in order to achieve natural development by enabling end users to develop
or modify interactive applications still using conceptual models, but with continuous
support that facilitates their development, analysis, and use [1].

Model-based languages are utilized at design time to help the user interface
designer cope with the increasing complexity of today's applications and contexts.
The underlying user interface models are mostly used to generate a final user interface
code, which is then executed at run time. However, approaches utilizing the models at
run time are receiving increasing attention. We agree with Sottet et al. [13], who call
for keeping the models alive at run time to make the design rationale available.

In the following, we present some research work that exploits model-based
approaches for multi-device ubiquitous applications. We show a new tool for desktop-
to-mobile adaptation, called customizable two-dimensional semantic redesign. We
present its underlying algorithm and compare its results with those of other current
tools. The environment also allows end users to customize the adaptation process.
Lastly, some conclusions are drawn along with indications for future work.

Related Work

Various approaches are possible to support adaptation for mobile devices. Bickmore [2] proposed a classification into five categories: device-specific authoring (one version for each target device type), multiple-device authoring (one version, with subversions for the various targets, e.g. using different stylesheets), client-side navigation (adaptation is performed directly by the client), Web page filtering (adaptation is obtained by content filtering) and automatic re-authoring (one version exists, which is then automatically adapted for the target device). Automatic re-authoring can be further divided into transducing (the original structure is preserved and the elements are adapted, e.g. images resized) and transforming (the structure is adapted as well). Our approach is an example of automatic re-authoring, supporting transforming (since the original pages can even be split into multiple mobile pages if they are too expensive in terms of space consumption).

Various contributions have been put forward in this area and it is not possible to
mention all of them. The OPA browser [14] allocates various functions for Web
browsing on each numerical key of a cellular phone. Buyukkokten et al. [4] proposed
a novel technique for form summarization, which is also able to automatically
summarise texts according to various policies. Laakko and Hiltunen [6] proposed a
technique for server side adaptation. We too support a solution using an adaptation
proxy but we also exploit logical descriptions that allow us to propose a more general
solution. The Roam system [5] is another environment for multi-device applications.
It also logically partition an application in a set of components but then it requires that
developers provide various implementations for different types of devices. Thus there
is little support for automatic adaptation. Studies on usability of mobile adaptation [7]
by Kaikkonen and Roto indicate that adaptation should not completely destroy the
original structure of the desktop pages in order to allow users to still be able to
associate the mobile pages with the original ones. One important issue in this
adaptation process is how to handle table adaptation. In [10] there is a proposal that
allows users to interactively fold and unfold the tables rows and/or columns.
However, such manual adaptations are lost when users access the tables again.

A Model-based Architecture for User Interface Adaptation

We have designed and developed a model-based architecture for user interface
adaptation, which supports reverse and forward transformations that are able to
transform existing desktop Web applications for various interaction platforms. The
basic assumption is that there exists a huge amount of easily accessible content for
desktop Web applications, which can be processed and transformed to support multi-
device interfaces, even across non-Web implementation languages. The advantage of
this solution with respect to others (e.g. [9]) is that it does not require that the
applications be implemented using a particular toolkit in order to make them able to
adapt.

When the user accesses the application through an interaction platform other than
the desktop, the intermediate adaptation server (which includes a proxy server)
transforms its user interface by building the corresponding logical description and
using it as a starting point for creating the implementation adapted to the accessing
device (see Figure 1). Lastly, the user interface implementation for the target device is
generated.

The reverse engineering module analyses the content of the HTML and the
associated CSS files and builds the logical description of the desktop user interface,
which is provided as input to the adaptation module.

<figure><img src="image_36.png" /><figcaption>Fig. 1. The Main Phases of the Adaptation Process</figcaption></figure>

In the process of creating an interface version suitable for a platform different from
the desktop, we use a semantic redesign module. This part of the environment
automatically transforms the logical description of the desktop version into the logical
description for the new platform. Therefore, the goal of this transformation is to
provide a description of the user interface suitable for the new platform. This means
that intelligent rules are used for adapting the description of the user interface to the
new platform taking into account its capabilities (e.g. using interface elements that are
more suitable for the new platform) but ensuring at the same time that the support for
the original set of tasks is maintained. This solution allows the environment to exploit
the semantic information contained in the logical description. In this case the semantic
information is related to the basic tasks that the user interface elements are expected
to support.

This software architecture for user interface adaptation currently uses MARIA
[12], a recent model-based language, which allows designers to specify abstract and
concrete user interface languages according to the CAMELEON Reference
framework [3]. This language represents a step forward in this area because it
provides abstractions also for describing modern Web 2.0 dynamic user interfaces and
Web service access. It provides an abstract language independent of the interaction
modalities and concrete languages for a number of platforms. In general, concrete
languages are dependent on the typical interaction resources of the target platform but
independent of the implementation languages.

In MARIA an abstract user interface is composed of one or multiple presentations, a data model, and a set of external functions. Each presentation contains a number of user interface elements (interactors) and interactor compositions (indicating how to group or relate a set of interactors), a dialogue model describing the dynamic behaviour of such elements, and connections indicating when a change of presentation should occur. The interactors are classified in abstract terms: edit, selection, only_output, control, interactive description, .. Each interactor can be associated with a number of event handlers, which can change properties of other interactors or activate external functions.

The Adaptation Transformation

We have designed a new tool for adaptation: *Customizable Two-dimensional Semantic Redesign*. It supports adaptation from desktop-to-mobile devices and overcomes some of the limitations of previous approaches in the area [11] because it allows users to configure the adaptation process and provides more control over costs calculation and the adaptation results. For example, while previous solutions calculated the screen space requested by the user interface elements mainly in terms of vertical extension, the new algorithm calculates both the horizontal and the vertical consumption of screen space.

The new algorithm takes as input the concrete description of a desktop user
interface in the MARIA language and goes through a number of steps. First, it
performs some basic transformations: if the user provides preferences regarding the
minimum and maximum fonts for the target device then the system transforms all the
textual content in order to fit it into the given range. Next, it calculates the cost of all
the interactors and composition operators in the provided specification. If the
resulting total cost is sustainable for the target device, then the corresponding logical
description is generated, otherwise it starts the process to reduce the cost in order to
make it sustainable. The basic elements are adapted for the target device first: the
images are shrunk, while preserving their aspect ratio, some interactors are replaced
with others that are semantically equivalent but need less screen space (e.g. a list can
be replaced with a drop-down menu), long texts are reduced in such a way that the
part exceeding a limit is shown only on request, image and text in tables are reduced
in size. After these basic transformations the overall cost is recalculated and if it is not
yet sustainable by the target device then the part of the algorithm related to page
splitting is activated. The purpose of this phase is to split the original desktop
presentation into two or more presentations that are sustainable for the target mobile
device. For this purpose the algorithm considers the interactor compositions
(groupings of elements or relations that involve two groups) and tables of elements,
and associates some of them to newly generated mobile presentations, removing them
from the current presentation in order to decrease its overall cost.

<figure><img src="image_37.png" /><figcaption>Fig. 2 The adaptation algorithm</figcaption></figure>

The elements that determine the cost of the interactors are: the font attributes (size, style, type), the vertical and horizontal space required by a text, image dimensions, interline value, interactor type, ...

The algorithm has a parameter (Scrolling to Avoid), which allows the specification of which scrolling (vertical or horizontal) to avoid in the case that the presentation cost exceed the limits in both directions.

When the splitting part is activated the algorithm looks for a structured element in
the logical description whose cost is sufficiently high that removing it would make
the presentation sustainable for the target device. Then, such structured element
would be allocated to a newly generated mobile presentation, which would be
accessible through a link inserted in the original one. The structured elements
considered are groupings, relations, data tables and layout tables. When the element candidate for removal is a data table, the splitting is implemented differently. The table is split into two parts, the part composed of the columns visible without horizontal scrolling remains in the original presentation with an additional link allowing the user to continue to browse it in a separate presentation containing the remaining columns, from which it is possible to return by a similar link.

<figure><img src="image_38.png" /><figcaption>Fig. 3. Example of table splitting</figcaption></figure>

In particular, the tool supports two ways to determine how splitting should be performed. In both cases it analyses the cost of the structured elements, which includes those of the composed interactors, and the cost of the tables (both data and layout tables). Then, the decision of the set of elements to allocate to the newly generated mobile presentation is given in one case by the most expensive element. In the other case the algorithm first calculates the elements whose removal would make the current presentation sustainable by the target device, and then selects the one that has the lowest cost. The rationale for this second option is that it allows obtaining a sustainable presentation but by removing the least amount of information possible, thus preserving the original design as much as possible.

## End-User Adaptation Customization

In the research on user interface adaptation, one issue that we are considering is how to provide users with more control over the adaptation process in order to improve the usability of its results. In this context more control can mean various things. One important aspect is control over the rules that drive adaptation to the
various platforms (the most common case is desktop-to-mobile adaptation). For example, the adaptation engine is able to split the desktop pages when they require considerable amounts of interaction resources but some users may like to have more control over the splitting algorithm. End-User Development [8] (EUD) can be defined as a set of methods, techniques, and tools that allow users of software systems, who are acting as non-professional software developers, at some point to create, modify or extend a software artefact. End-users already have difficulties with single device applications, thus it easy to understand how such difficulties increase when considering applications for multi-device environments. This is one further reason for providing better support for EUD in ubiquitous applications.

Figure 5 and 6 show the user interface that allows end users to configure the adaptation process. The various parameters are grouped according to the related user interface aspect considered. For the fonts, it is possible to specify the minimum and maximum font size in the target device, and the associated measure unit. For the radio buttons it is possible to indicate whether they should be transformed into an interactor that supports the same semantics but using less space screen. In this case, it is possible to specify the threshold, in terms of number of choice options, which should trigger the transformation and the type of interactor to use for its replacement. Similar parameters are available for the list boxes. Other parameters concern the maximum number of characters for texts, maximum and minimum dimensions for images. These parameters determine the cost of rendering a presentation. This cost is compared with the overall sustainable cost in the target device, which is given by the screen resolution multiplied by the horizontal and vertical tolerance. The higher the tolerance coefficient values are, the more scrollable the generated user interface will be. This means that end users have the possibility to specify to what extent the adapted content will be scrollable in the target device. The table tolerance provides an additional factor to consider when calculating the sustainable cost. In practise, this means that when there are tables, more scrolling will be acceptable before deciding to split the presentation.

<figure><img src="image_39.png" /><figcaption>**Fig. 4.** An example application: Wikipedia.</figcaption></figure>
Figure 4 shows the structure of the user interface of a well-known application, Wikipedia, and next we see how the splitting changes depending on the customization parameters specified. In next Figures we show two example configurations, which mainly differ for the scrolling to avoid parameter (in one case is vertical and in the other is horizontal) and the coefficients for display tolerance (in one case they are 20 and 80, in the other one they are 20, 500).

<figure><img src="image_40.png" /><figcaption>Fig. 5. First Example of Adaptation Configuration and Associated Results.</figcaption></figure>
The customization interface also allows the user to set the priority of the type of
scrolling (horizontal or vertical) to avoid and the algorithm splitting policy to apply.
In this way, we obtain the specification of user preferences regarding adaptation,
which can also be reused for other applications more easily than solution such as
collapse-to-zoom [15], where the user can express preferences only associated to a
given application.

<figure><img src="image_41.png" /><figcaption>**Fig. 6.** Second Example of Adaptation Configuration and Associated Results.</figcaption></figure>
Then, we can see for each configuration the resulting adapted mobile pages. In the first case the main page is split into three mobile pages (Figure 5). In the first mobile presentation we have highlighted the automatically generated links to the other mobile pages. In the second case (Figure 6), only two pages are generated from the splitting. This is because in order to fit with the vertical scrolling was sufficient to cut only one big element, which referred to the main central content part.

Please note that the results of the adaptation applied to Web sites such as Wikipedia can change depending on the change of the actual content, which continuously varies in such sites.

## An Example Application

In order to better understand how our approach works we can consider an example desktop Web application (see Figure 7). For the sake of clarity we do not use a particularly complex example.

<figure><img src="image_42.png" /><figcaption>Fig. 7. An example user interface</figcaption></figure>

When the reverse engineering module performs the analysis of this page code, it builds the corresponding logical description (which is highlighted in the Figure). At the first level it identifies a group (G1) associated with the header, a group (G2) associated with the central part, and one group (G3) mainly associated with the data
table. Lastly, a final long text at the bottom is identified. Recursively it identifies the
elements included in each group. The top group is composed of an image and some
text, the central group is composed of an image and a form, the bottom group is
composed of text and a data table. The form is then composed of a number of
interactive elements and texts. Now, let us assume that the following parameters have
been specified to configure the adaptation process:

Minimum font size = 10px
Maximum font size = 18px
Max image width = 200px
Max image height = 150px
Horizontal tolerance = 10%
Vertical tolerance = 10%
Radio button transformation = yes
Radio button threshold = 3
Radio button mapping = drop down list
Scrolling to Avoid = horizontal
Interactor composition to cut = highest
Long text limit = 300

According to the algorithm previously described, first some basic textual content
adaptation is performed. For example, the text "Flight information crawler",
contained in Grouping G1, has a size (33px) greater than the value specified in the
parameter maximum font size, and consequently is reduced to this limit.

Then, the algorithm calculates the costs in terms of screen consumption of the
basic interface elements, and then consequently calculates the costs of the higher
elements in the logical structure.

<figure><img src="image_43.png" /><figcaption>Fig. 8. The costs of the example</figcaption></figure>

Figure 8 shows the resulting costs. For each element a pair of values is provided
indicating both the horizontal and the vertical costs. If we consider the specified
values for horizontal and vertical tolerance and the resolution of the current device
(360x480 pixels), the maximum sustainable horizontal cost would be 396 pixel, and the vertical 528 pixels. If we look at the overall page cost, given by the cost of G0, we can notice that it is higher than the sustainable cost and consequently the adaptation transformation should move on to the next phase, which involves adaptation of the user interface elements. In particular, in this case we have:

* *The transformation of long texts*, since G0 contains a text longer than 300 characters, the text is split into two parts, one reachable only on request through a link;

* *The transformation of images*, the image contained in the G2 grouping is larger than the limits indicated by max image width and max image height, thus it is scaled from 343x248 pixel to a resolution of 198x143 pixels.

* *Conversion into equivalent interactors*, the radio buttons (an example is the interactor SC in the Figure) are transformed according to the adaptation parameters that indicate that radio buttons be converted into drop-down menus when there are more than three options.

* *Reduction of space taken up by data tables*, the data table DT, contained in the Grouping G3, is reduced by decreasing the size of all the texts contained in it.

Figure 9 shows the updated costs in the user interface versions with the elements adapted as described. It is possible to note that even the resulting overall cost is still too large for the target device. Thus, the phase dedicated to page splitting is activated.

<figure><img src="image_44.png" /><figcaption>Fig. 9. The updated costs of the example</figcaption></figure>

As described previously the splitting algorithm is driven by two parameters: Scrolling to Avoid and Interactor composition to cut. In our example the first one is set to *horizontal*, and the second one to *highest*. According to these values, the splitting algorithm looks for the element with the highest cost, which is suitable to
avoiding horizontal scrolling. In this case it is the data table DT in Figure 9. According to the rules previously introduced the table is split in such a way as to allocate to a newly generated mobile presentation the portion exceeding the horizontal limit. Thus, at the end of the first cycle the algorithm produces two newly generated additional mobile presentations: one for the excessive table portion and one for the excessive text (see Figure 10).

<figure><img src="image_45.png" /><figcaption>**Fig. 10.** The presentation generated by the semantic redesign</figcaption></figure>

## Comparison with Other Approaches

We have conducted a study comparing our tool, in terms of adaptation results, with two publicly available tools for desktop-to-mobile adaptation: Mowser (http://mowser.com) and Skweezer (http://www.skweezer.com). Figure 11 shows an example form interface adapted using the three systems.
<figure><img src="image_46.png" /><figcaption>Fig. 11. Form adaptation comparison</figcaption></figure>

By comparing the three versions we have noted that Mowser resizes only the images larger than 150 pixels, ignores style sheets and text attributes indicated in the pages because it associates them with predefined sheets. It provides no particular support for long texts, tables, or change of interactors. In addition, it aims to reduce vertical use of screen space, but this is obtained by requiring users to perform considerable horizontal scrolling.

Squeezer follows a different policy. It reduces the image quality but it does not change their dimensions. Like Mowser, it ignores the colours and the properties specified by the style sheets but it preserves some HTML tags ( <b>, <i> and <u> ) for the text formatting. Also Squeezers does not support long text transformations, table management (see Figure 12), or interactor changes. Differently from Mowser, Squeezer aims to reduce horizontal scrolling, which implies increasing the vertical one. It also aims to reduce the page download time by reducing the size of its content in terms of bytes.
<figure><img src="image_47.png" /><figcaption>Fig. 12. Table adaptation comparison</figcaption></figure>

The results of this comparison were encouraging because our tool has shown to be more flexible. Indeed, it allows end users to customize the adaptation parameters and is able to adapt a higher number of interface element types than the other two tools (e.g. tables and long texts do not receive specific adaptation transformations with the other two tools).

Conclusions and Future Work

Ubiquitous environments call for adaptive systems in order to adapt to the varying interaction resources. Model-based approaches can provide useful support in this context. We propose a solution for desktop-to-mobile adaptation of Web user interfaces, which overcome limitations of previous ones.

The solution is able to dynamically handle Web pages and build the corresponding logical description through a reverse engineering module able to analyse all the HTML and CSS associated tags. In the adaptation interface elements can be replaced with others that are semantically equivalent but require less screen space. The scripts are preserved in the adapted version. Content such as Flash, Java applets are not currently adapted.

In addition, there is a need for providing users with more control on ubiquitous interfaces, according to the end-user development paradigm. In this paper we have presented a solution that also allows end-users to customize the desktop-to-mobile adaptation in order to change the results that can be obtained by automatic user interface generation.

We plan to further extend this work in various directions. The customization user interface can be improved in order to make the effects of the various customization parameters more understandable. In addition, in this work we have considered only desktop-to-mobile adaptation but other types of transformations can benefit from the approach proposed, e.g. graphical-to-vocal adaptation.
Acknowledgments

This work has been partially supported by the EU ICT STREP Project OPEN
(http://www.ict-open.eu/)

References

1. Berti, S., Paternò, F., Santoro C., "Natural Development of Ubiquitous Interfaces", Communications of the ACM, September 2004, pp.63-64, ACM Press.

2. Bickmore T. et al. Web page filtering and re-authoring for mobile users. Computer Journal special issue on Mobile Computing, vol. 42(no. 6):pp. 534-546, 1999.

3. Calvary, G., Coutaz, J., Bouillon, L., Florins, M., Limbourg, Q., Marucci, L., Paternò, F., Santoro, C., Souchon, N., Thevenin, D., and Vanderdonckt, J. 2002. The CAMELEON reference framework. CAMELEON Project. Deliverable 1.1

4. Buyukkokten O., Kaljuvee O., Garcia-Molina H., Paepcke A., Winograd T. ,. Efficient web browsing on handheld devices using page and form summarization. TOIS, pages 82-115, 2002.

5. Chu H., Song H., Wong C., Kurakake S. and Katagiri M., Roam, a seamless application framework, Journal of Systems and Software, Volume 69, Issue 3, 2004, pp. 209-226, Elsevier.

6. Laakko T. Hiltunen T. Adapting web content to mobile user agents. IEEE Internet Computing, Vol. 9(no. 2):46-53, March-April 2005.

7. Kaikkonen A. Roto, V. Perception of narrow web pages on a mobile phone. Proc. Human Factors in Telecommunications, 2003.

8. Lieberman, H., Paternò, F., Wulf W. (eds), End-User Development, Springer Verlag, ISBN-10 1-4020-4220-5, 2006.

9. Melchior, J., Grolaux, D., Vanderdonckt, J., Van Roy, P., A Toolkit for Peer-to-Peer Distributed User Interfaces: Concepts, Implementation, and Applications, pp. 69.78, EICS'09, July 15-17, 2009, Pittsburgh, Pennsylvania, USA.

10. Ohnishi K., Tajima K.. Browsing large html tables on small screens., ACM Symposium on User Interface Software and Technology (UIST '08).

11. Paternò, F., Santoro, C., Scorcia A Automatically Adapting Web Sites for Mobile Access through Logical Descriptions and Dynamic Analysis of Interaction Resources. AVI 2008, Naples, May 2008, ACM Press, pp. 260-267.

12. Paternò F., Santoro C., Spano L.D., "MARIA: A Universal Language for Service-Oriented Applications in Ubiquitous Environment", ACM Transactions on Computer-Human Interaction, Vol.16, N.4, November 2009, pp.19:1-19:30.

13. Sottet J., Ganneau V., Calvary G., Coutaz J., Demeure A., Favre J., Demumieux R.: Model-Driven Adaptation for Plastic User Interfaces. INTERACT (1) 2007: 397-410.

14. Uemukai T., Nishio S., Y. Arase, T. Hara. Opa browser: a web browser for cellular phone users. ACM Symposium on User Interface Software and Technology (UIST '07), pages 71-80.

15. Wang-Wei-Ying C., Baudisch P., Xie X.. Collapse-to-zoom: Viewing web pages on small screen devices by interactively removing irrelevant content. ACM Symposium on User Interface Software and Technology (UIST '04), pages 91-94, October 2004.

16. Weiser M., "The Computer for the 21st Century" - Scientific American Special Issue on Communications, Computers, and Networks, September, 1991.
Extending UsiXML to support User-aware
Interfaces

Ricardo Tesoriero¹² and Jean Vanderdonckt¹

¹ Université catholique de Louvain
Place des Doyens, 1 (B-1348)
Louvain-la-Neuve, Belgium

² University of Castilla-La Mancha
Av. España S/N. Campus Universitario de Albacete (02071)
Albacete, Spain

ricardo.tesoriero@uclm.es, jean.vanderdonckt@uclouvain.be

**Abstract.** Mobile and portable devices require the definition of new user interfaces (UI) capable of reducing the level of attention required by users to operate the applications they run to improve the calmness of them. To carry out this task, the next generation of UIs should be able to capture information from the context and act accordingly. This work defines an extension to the UsiXML methodology that specifies how the information on the user is modeled and used to customize the UI. The extension is defined vertically through the methodology, affecting all layers of the methodology. In the Tasks & Concepts layer, we define the user environment of the application, where roles and individuals are characterized to represent different user situations. In the Abstract UI layer, we relate groups of these individuals to abstract interaction objects. Thus, user situations are linked to the abstract model of the UI. In the Concrete UI layer, we specify how the information on the user is acquired and how it is related to the concrete components of the UI. This work also presents how to apply the proposed extensions to a case of study. Finally, it discusses the advantages of using this approach to model user-aware applications.

# 1 Introduction

In 1994, Mark Weiser introduced the notion of Calm Technology in [16]. The aim of Calm Technology is to reduce the “excitement” of information overload by letting the user select what information should be placed at the center of their attention and what information should be peripheral. A way to support Calm Technology is the use of the context information to reduce users’ work.

According to [11], context-aware applications [4] are characterized by a hierarchical feature space model. At the top level, there is a distinction between human factors, in the widest sense, and the physical environment. Both, the human factors and the physical environment, define three categories of features each. While, the human factors are defined in terms of features related to (a) the
information on the user (knowledge of habits, emotional state, bio-physiological conditions, ...); (b) the user's social environment (co-location of others, social interaction, group dynamics, ...); and (c) the user's tasks (spontaneous activity, engaged tasks, general goals, ...); the physical environment are defined in terms of features related to (a) the location (absolute position, relative position, co-location,...), (b) the infrastructure (surrounding resources for computation, communication, task performance, ...) and (c) physical conditions (noise, light, pressure, ...).

This article explores the development of multi-modal UIs that are affected by human factors. Concretely, it focuses on those features related to the information on the user (i.e. emotional state, bio-physiological conditions, skills, experience, ...) and the user's tasks (i.e., defined by the role played in the society).

The proposal is based on an extension to the UsiXML [14] methodology based on the social model of CAUCE methodology defined in [13]. While UsiXML provides a model-based approach to design multi-modal UIs based on the Cameleon reference framework [2], the social model provides designers with the ability express how user features affect the application UI.

The paper is organized as follows. In Section 2, we present the most relevant related works on the development of multi-modal UIs. Then, the UsiXML extension to support user awareness is exposed in Section 3. Afterwards, the extension is applied to a case of study in Section 4. Finally, in Section 5 we expose conclusions and future works.

## 2 Related Work

Teallach tool and method [1] exploit three models: a task model, a domain model as a class diagram, and a presentation model both at logical and physical levels. Teallach enables designers to start building a UI from any model and maps concepts from different models one to each other. However, the tool does not support the development of context-aware UIs. Moreover, from the user modeling point of view, Teallach does not support any type of user profile nor awareness.

The approach exposed in [3] describes a toolkit of interactors, which are designed to develop UIs that handle both input and output using multiple mechanisms. The toolkit supports adaptation for a change in the resources available to the widgets, or a change in the context the platform is situated in. However, the approach does not support any model to capture user information from external sources that are not directly related to the platform or the widget contents defined in the interface.

XIML [5] is a more general UIDL than UIML as it can specify any type of model, any model element, and relationships between. The predefined models and relationships can be expanded to fit a particular context of use. The term context is interpreted as the platform of the application is running and not the application is adapter to such platform, instead of the information that affects,
or is relevant, to the application. No other issue related to the context awareness
is taken into account by the model.

SeescoaXML [8] supports the production of UIs for multiple platforms and
the run-time migration of the full UI. However, the development does not take
into account the definition of the user-aware behavior of the UI.

The CTTE (ConcurTaskTrees Environment)[9] is a development environ-
ment to support the development and analysis of task models for interactive
systems. Based on these models, the TERESA (Transformation Environment
for inteRactivE Systems representAtions) [10] produce different UIs for multiple
computing platforms by refining a general task model. Thus, various presenta-
tion and dialog techniques are used to map the refinements into XHTML code
adapted for each platform such as the Web, the PocketPC, and mobile phones.

Although CTT (the language used to describe the task model of the applica-
tions developed using TERESA) supports the role definition and the definition
of different task models for each role, the task and role concepts are so coupled
that the definition of similar interfaces derive in different models (one for each
role). Besides, this approach does not take into account the attributes of role def-
initions, although CTT allows designers to assign values to standard attributes.
Thus, the definition of custom attributes is not supported directly.

RIML [12] consists of an XML-based language that combines features of
several existing markup languages (e.g., XForms, SMIL) in a XHTML language
profile. This language is used to transform any RIML-compliant document into
multiple target languages suitable for visual or vocal browsers on mobile devices.
Although RIML provides the ability to specify multi-modal UI, RIML is focused
on the view of the application independently of the context it is being executed.
Therefore, no context or user awareness is taken into account.

3 The user-aware UsiXML extension

UsiXML defines a development process based on the Cameleon Reference Frame-
work [2] to build multi-device interactive applications. The development process
is divided into four layers.

The Task & Concepts (T&C) layer describes users’ tasks to be carried out,
and the domain-oriented concepts required to perform these tasks.

The Abstract User Interface (AUI) defines abstract containers and individ-
ual components [7] (two forms of Abstract Interaction Objects [15]) by grouping
subtasks according to various criteria (e.g., task model structural patterns, cog-
nitive load analysis, semantic relationships identification), a navigation scheme
between the container and selects abstract individual component for each con-
cept so that they are independent of any modality. Thus, an AUI is considered
as an abstraction of a CUI with respect to interaction modality.

The Concrete User Interface (CUI) concretizes an abstract UI for a given
context of use into Concrete Interaction Objects (CIOs) [15]. It also defines
the widget layouts and the interface navigation. It abstracts a FUI into a UI
definition that is independent of any computing platform. Therefore, a CUI can
Both, the user feature level and the user profile level are rooted in the User-Model metaclass, as depicted in Figure 2. It represents the user characteristics that affect the UI at both levels of abstraction.

<figure><img src="image_48.png" /><figcaption>Fig. 2. The UserModel metaclass</figcaption></figure>

The user feature level defines the user characteristics in terms of *Roles* and *Features*. We have defined *Role* as a set of user *Features* where a *Feature* represents an attribute related to the user playing this role in the system. For instance, a *Patient Role* may define the age, cardiac pulse, temperature and glucose level *Features*.

In order to represent *Roles* that have common *Features*, we have defined the *RoleSpecialization* relationship between *Roles*. It is a directional relationship defined by the *source* and the *target* attributes. The semantic meaning of the *RoleSpecialization* relationship is defined as follows:

Let A and B be instances of the *Role* metaclass; Let $FwoS(R)$ be the function that takes the *Role* R as parameter and returns the set of *Features* defined by R *Role without taking into account the RoleSpecialization relationship*. Let $F(R)$ be the function that takes the *Role* R as parameter and returns the set of *Features* defined by R *Role taking into account the RoleSpecialization relationship*. Finally, let $S(A, B)$ be the *RoleSpecialization* relationship that defines A as the *source* and B the *target* of the relationship.

Then,

$$F(B) = FwoS(B) \\ F(A) = FwoS(B) \cup FwoS(A)$$

The user profile level defines user characteristics in terms of *Individuals* and *FeatureConstraints*

Users are characterized according to the *Features* defined by the *Roles* they play in the system. The characterization at this level is defined by the *Feature-Constraints* metaclass that is related to an *Individual* playing a *Role*. Thus, the *RoleRealization* metaclass defines the relationship between the *Role* and the *Individual*. Then, the *Individual* defines a set of *FeatureConstraints* that are related to the *Features* defined by the *Role* it plays in the system.
For instance, following the example exposed on the user feature level, the
Patient *Role* may be realized by the *aPatientWithFever Individual* that defines
a *FeatureConstraint* where the *temperature Feature* is higher than 38 Celsius
Degrees.

Thus, the same user may be a *aPatientWithFever* or not, according to the
body *temperature* of the user.

Finally, the *UserModel* is related to the *TaskModel* by the means of reflecting
how the characteristics defined by the *UserModel* are propagated to the UI in
the T&C layer of the UsiXML methodology. Each *Task* defined in the *TaskModel*
is affected by an *Individual* that describes the user situation in which the *Task*
is performed. Therefore, in order to perform a *Task*, all the *FeatureConstraints*
defined by the *Individual* that is related to the task must be satisfied.

## 3.2 The AUI layer extension

The AUIModel is part of the AUI layer of the UsiXML methodology. Although
the user awareness extension does not affect the AUIModel definition directly,
it introduces some modifications in the AUI layer by the means of the definition
of new inter-model relationships in the MappingModel.

These relationships are established between Abstract Interaction Objects (*AIOs*) and *Individuals*. However, they do not affect Abstract Individual Components *AICs* in the same way they affect Abstract Containers *ACs*.

On the one hand, *ACs* are directly affected by *Individuals*. On the other
hand, *AICs* are affected through *Facets* that are affected by *Individuals*.

The Figure 3 shows the extensions of the MappingModel and how it affects
the AUIModel.

<figure><img src="image_49.png" /><figcaption>Fig. 3. MappingModel extensions in the AUI layer</figcaption></figure>
Let suppose that an *Individual* is related to an *AC*. If all the *FeatureConstraints* of the *Individual* are satisfied, then the *AC* is “active”, “enabled” or “available”. Otherwise, the *AC* is “inactive”, “disabled” or “unavailable”.

*AICs* define *Facets*. These *Facets* are manipulated by the *FeatureConstraints* defined by the *Individual* they are attached to.

Let suppose that an *Individual* is related to a *Facet*. If all the *FeatureConstraints* of the *Individual* are satisfied, then the *Facet* is “active”, “enabled” or “available”. Otherwise, the *Facet* is “inactive”, “disabled” or “unavailable”.

Therefore, the behavior of the UI is affected by profiles defined in the Tasks & Concepts layer of the UsiXML methodology. This relationship is defined by the *FacetObserves* and *ContainerObserves* submetaclasses of the *Observes* metaclass, which belongs to the MappingModel.

Thus, as an *AC* or a *Facet* can “observe” many *Individuals*, a conflict among *FeatureConstraints* may arise. Therefore, we enable the *Facet* or *AC* when any of the *Individuals* match the user state.

Depending on the development path, these relationships can be derived from the TaskModel and the *IsExecutedIn* inter-model relationships defined in the MappingModel using transformation rules defined in the TransformationModel.

## 3.3 The CUI layer extension

The extension to support the user awareness in the CUI layer is depicted in Figure 4.

<figure><img src="image_50.png" /><figcaption>Fig. 4. Relating Sensors and Interpreters to Features</figcaption></figure>

The metamodel is divided into two parts:

- The CUI model extension

- The mapping model extension
The CUI extension goal is the description of the system is aware of the information on the user. To carry out this task, we have introduced the *Sensor* and the *Interpreter* entities that are in charge of capturing, interpreting and providing information to be processed by the system.

The information can be perceived in different ways according to the sensor it is used to capture it. Therefore, we have defined two types of *Sensors*: *EventDrivenSensors* and *PollingSensors*. The information that is perceived by sensors should be interpreted accordingly. To perform this task, we have defined *Interpreters*. *Interpreters* are in charge of transforming information that comes from *Sensors* into information that is compatible with the user environment definition.

We have defined three types of *Interpreters*: the *PollingInterpreter* that deals with information coming from a *PollingSensor*, the *EventDrivenInterpreter* that is in charge of interpreting information from an *EventDrivenSensor*, and the *ComposedInterpreter* that is responsible for the interpretation of information coming from several sensors. The *Interpreter* hierarchy is an implementation of the Composite design pattern [6].

The information processed by the CUI extension is propagated through the rest of the model by the mapping model extension. Therefore, the connection between the *CIOs* and the rest of the models is performed through the *interpreterUpdates* submetaclass of the *intermodelRelationship* defined by the MappingModel. This relationship is used to define the relationship between *Features* defined in the user environment model and the *Interpreters*.

Thus, *Individuals* are notified of the changes of the information on the user through *FeatureConstraints*.

## 3.4 The transformation process and the FUI

In this section, we point out some issues related to the transformation process that takes place between the abstract user interface *AUI* and the concrete user interface *CUI*. The way the information captured by the models is translated to source code depends on the *AIO* we are dealing with.

If we are dealing with *abstractContainers*, the translation of an *Individual* that matches the state of the user usually results in the modification of a property in the *CIO* that represents it. For instance, the *visible*, *enabled* or *opaque* property of the *CIO* is set to *true*.

However, the mechanism used by *AICs* is not the same because *Individuals* are related to *Facets*. Therefore, some aspects of the *AICs* may match some *Individuals* and some of them may not.

Suppose that an *AIC* defines the following *Facets*: *Input*, *Output* and *Navigation*. In addition, each *Facet* is related to different *Individuals*: $I_1 \leftrightarrow Input$, $I_2 \leftrightarrow Output$ and $I_3 \leftrightarrow Navigation$.

Thus, if $I_1$ is the unique *Individual* that matches the user environment, then the *AIC* may be represented by a *iguideField*. However, if $I_2$ is the unique *Individual* that matches the user environment, then the *AIC* may be represented by a
**Label.** Finally, if $I_3$ is the unique *Individual* that matches the user environment, then the *AIC* may be represented by a **Button** or a **Link**.

Therefore, the same *AIC* can be turned into different *CIOs*, at runtime, according to the user state. The Figure 5 represents the possible results of the transformation, according to the interface specification taking into account the user environment.

The situation is solved using different mappings for the same *AIO*. Thus, the user state defines which component is available according to the user environment.

<figure><img src="image_51.png" /><figcaption>Fig. 5. Possible results on the Final User Interface</figcaption></figure>

# 4 Case of study

The application that will serve as case of study is the “Healthy Menu”. The goal of the application is the presentation of dishes according to the role and the biophysical state of the user. Thus, views and controls are customized to provide the user with Calm UIs.

The explanation is focused on issues related to the definition of the user environment and the relationships between this environment and the UI. Other issues are left behind for the sake of clarity.

## 4.1 The user model

The application defines five roles: Users, Patients, Doctors, Nurses and Visitors.

The Users of the application are identified by the *idNumber*, *roleName* and *user Name* features.

As all other roles are specializations of the User role, these roles inherit User features. Besides these features, Patients are characterized by the *age*, *temperature* and *glucose* features; Doctors by the *specialty*; Nurses by the *experience* (in years); and Visitors by the patient they are visiting (the *patientId* feature).

As we will focus on the Patient role, we have defined five *Individuals* for this role (aPatient, NormalPatient, PatientWithHyperGlycemia, PatientWithHipoGlycemia and PatientWithFever); and only one for each remaining role (anUser, aPatient, aNurse, aDoctor and aVisitor).
Each *Individual* is defined by a set of *FeatureConstraints*. For instance, we say that a *Patient* is normal, if the body temperature is between 36.5 and 38 Celsius degrees and the Sugar level in blood is between 70 and 110 millimoles/liter.

The Figure 6 shows the user model of the application. Rounded rectangles represent Roles, the dashed rectangles defined inside them represent features, the specialization of roles is represented by a triangle-ended arrow pointing to the role that is being specialized, circles represent instances and the rectangles on drawn on the right of these circles represent feature constraints.

<figure><img src="image_52.png" /><figcaption>Fig. 6. Healthy screen user model</figcaption></figure>

## 4.2 The task model

The simplified version of the task model for the Healthy screen application is defined in Figure 7.

<figure><img src="image_53.png" /><figcaption>Fig. 7. Healthy screen task model</figcaption></figure>
The *UserLogin* task allows the system to identify the user that is using the application. The information that is retrieved from the system is exposed to the user by the means of the *ShowInfo* system task. Then, the user is able to choose the menu (*SelectMenu* task). To improve task model readability, the menu is divided into three types of selection defined under the *SelectDrink*, *SelectMeal*, *SelectDessert* task. Each task represents a possible menu option. For instance, *SelectWater* (to select water as a drink) is available to all users. However, *SelectWine* (to select wine as a drink) is available for normal patients only. It is also possible to assign more than one individual for each task. Therefore, it is available if any of the involved individuals match the “user state”.

## 4.3 The AUI model

Once the user model was defined, we define the *AUI* model. The AUI model defines the UI without taking into account its modality.

Although the AUI layer introduced two new types of mappings to describe the relationship between *AIOs* and *Individuals*, the extension does not introduce new elements to AUIModel. Therefore, AUIs are described in the same way they are described traditionally.

The Figure 8 depicts a partial view of the *AUI* model focused on the *Patient Role*. On the left, we show an overview of the containers that are related to the roles defined in the user environment. On the right, we show a detailed view of the Patient role AUI.

<figure><img src="image_54.png" /><figcaption>Fig. 8. Partial Healthy Menu AUI model</figcaption></figure>

Then, these *AUIs* are linked to *Individuals* to reflect the modifications in the “user state” accordingly.

The Table 1 shows the *Observes* relationships that relate *ACs* and *Facets* to *Individuals*. These relationships are defined in the mapping model and described in terms of the $O_c(I, AC)$ and the $O_f(I, F)$ functions. While the $O_c$ function represents the *ContainerObserves* relationship, which relates an *Individual I* to an *AbstarctContainer AC*, the $O_f$ function represents the *FacetObserves* relationship, which relates an *Individual I* to a *Facet F*.
## AUI mappings

<table><caption>**Table 1.** AUI mappings</caption><thead><tr><th colspan="2">Containers</th></tr></thead><tbody><tr><td>O<sub>c</sub>(aPatient, PatientMenu)</td><td>O<sub>c</sub>(aNurse, NurseMenu)</td></tr><tr><td>O<sub>c</sub>(aDoctor, DoctorMenu)</td><td>O<sub>c</sub>(aDoctor, DoctorMenu)</td></tr><tr><td>O<sub>c</sub>(aVisitor, VisitorMenu)</td><td>O<sub>c</sub>(aVisitor, VisitorMenu)</td></tr></tbody><thead><tr><th colspan="2">Facets</th></tr></thead><tbody><tr><td>O<sub>f</sub>(anUser, nAccept)</td><td>O<sub>f</sub>(anUser, oIdNumber)</td></tr><tr><td>O<sub>f</sub>(anUser, cAccept)</td><td>O<sub>f</sub>(anUser, oDesease)</td></tr><tr><td>O<sub>f</sub>(anUser, nReject)</td><td>O<sub>f</sub>(anUser, nReject)</td></tr><tr><td>O<sub>f</sub>(anUser, cReject)</td><td>O<sub>f</sub>(anUser, cReject)</td></tr><tr><td>O<sub>f</sub>(aPatientWithHipoGlycemia, oDrink)</td><td>O<sub>f</sub>(aPatientWithFever, iDrink)</td></tr><tr><td>O<sub>f</sub>(aPatientWithHipoGlycemia, oMeal)</td><td>O<sub>f</sub>(aPatientWithFever, oMeal)</td></tr><tr><td>O<sub>f</sub>(aPatientWithHipoGlycemia, oDessert)</td><td>O<sub>f</sub>(aPatientWithFever, iDessert)</td></tr><tr><td>O<sub>f</sub>(aPatientWithHyperGlycemia, oDrink)</td><td>O<sub>f</sub>(aNormalPatient, iDrink)</td></tr><tr><td>O<sub>f</sub>(aPatientWithHyperGlycemia, oMeal)</td><td>O<sub>f</sub>(aNormalPatient, iMeal)</td></tr><tr><td>O<sub>f</sub>(aPatientWithHyperGlycemia, oDessert)</td><td>O<sub>f</sub>(aNormalPatient, iDessert)</td></tr><tr><td>O<sub>f</sub>(aPatientWithFever, oDrink)</td><td>O<sub>f</sub>(aNormalPatient, oDrink)</td></tr><tr><td>O<sub>f</sub>(aPatientWithFever, oMeal)</td><td>O<sub>f</sub>(aNormalPatient, oMeal)</td></tr><tr><td>O<sub>f</sub>(aNormalPatient, oDessert)</td><td>O<sub>f</sub>(aNormalPatient, oDessert)</td></tr></tbody></table>

To conclude this section, we analyze the *drink AIC* defined in Figure 8 to show an example of how to define the AUI and how to link it to the *Individuals* defined in the user model.

From the AUI perspective, the *drink AIC* represents the component that is in charge of providing users with the information about drinks. This information may be input and output information (if the user is able to select the drink, i.e. *aNormalPatient*), or may be output information only (if the user is not able to select the drink, i.e. *aPatientWithHyperGlycemia*). Therefore, two Facets (*oDrink* and *iDrink*) were defined for this *AIC*.

From the Mapping perspective, lets analyze the individual *aPatientWith-HipoGlycemia* and its relationship with the *drink AIC*. The *O<sub>f</sub>(aPatient With Hipo Glycemia, oDrink)* is the only relationship between individuals and facets of the *drink AIC*. Therefore, the *drink AIC* is an output control. However, if we analyze the *aNormalPatient* individual, we see that the *O<sub>f</sub>(aNormal Patient, iDrink)* and the *O<sub>f</sub>(aNormal Patient, oDrink)* relationships define an input/output relationship.

## 4.4 The CUI model

The *CUI* definition is based on two sensors: *PollingGlucoseSensor* and *Polling TemperatureSensor*. Both of them are instances of the *PollingSensor* metaclass. It also defines two instances of the *PollingInterpreter* metaclass, the *GlucoseInterpreter* and *TemperatureInterpreter*.
Table 2 shows the mapping between the Patient role features and sensors. The function $R_p(S_p, I_p)$ represents the relationship between a PollingSensor ($S_p$) and a PollingInterpreter($I_p$). The function $U(I, F)$ represents an instance of the InterpreterUpdates subaclass of Updates, which relates an Interpreter to a Feature.

<table><caption>**Table 2.** CUI mappings</caption><thead><tr><th colspan="2">CUI mappings</th></tr></thead><tbody><tr><td colspan="2">Sensors</td></tr><tr><td>R<sub>p</sub>(PollingGlucoseSensor, GlucoseInterpreter)</td><td>R<sub>p</sub>(PollingTemperatureSensor, TemperatureInterpreter)</td></tr><tr><td colspan="2">Features</td></tr><tr><td>U(GlucoseInterpreter, Patient.glucose)</td><td>U(TemperatureInterpreter, Patient.temperature)</td></tr></tbody></table>

To illustrate the use of the elements defined in the CUIModel, we will expose how the *temperature* feature is related to the environment.

The temperature is captured from the environment through a *Polling Temperature Sensor*. To poll the sensor, we have to link it to a *Polling Temperature Interpreter* in charge of requesting the sensor status and propagating it to the rest of the system.

Finally, the *Polling Temperature Interpreter* is linked to the *temperature* feature of the Patient role to propagate the changes from the environment.

## 4.5 The HealthyScreen FUI

The result of the *FUI* of a Patient is depicted in Figure 9. The first capture shows the UI for a Patient whose vital parameters are considered normal, the second one shows the UI for a Patient whose body temperature is above normal, and the third capture shows the UI for patients affected with HipoGlycemia or HyperGlycemia.

Finally, we show how the elements defined by the user awareness extension work together in this case of study. To see the effect of the user awareness, we set the initial state of the individual as a *Normal Patient*. Then, we modify the *temperature* feature of the individual, and we set it to 39 Celsius degrees.

This change in the temperature is captured by the *Polling Temperature Interpreter* that is constantly polling the *Polling Temperature Sensor* to be aware of the changes in the environment (in this case the user). Once the interpreter captures the information, it is propagated to the *temperature* feature defined by the Patient role. All features are linked to the features constraints that reference them. Thus, the chance is captured by the individual that is defined by these constraints. If all the feature constraints that are defined by the individual are satisfied then the individual is able to propagate this information through the
<figure><img src="image_55.png" /><figcaption>Fig. 9. Healthy Menu GUIs</figcaption></figure>

*IndividualObserves* mappings to the AUI. Consequently, the UI is aware of the
changes produced by the change on the temperature level.

5 Conclusions and Future Work

This work exposes a model-based approach to develop user-aware multi-platform
and multi-modal UIs based on the UsiXML methodology. It encourages the
separation of the user modeling from the application domain to improve the
model reuse during the development of UIs.

The approach embraces all steps of the application. It means that covers from
conceptual modeling of the user environment to the specification of the sensing
infrastructure to support the different user profiles dynamically.

In the “Tasks & Concepts” step of the methodology, we introduced two lev-
els to define custom characterizations of the users. While the first level allows
designers to specify the user features that are taken into account by the appli-
cation, the second one allows designers to quantify these characteristics in order
to characterize a group of individuals that have common characteristics.

As consequence, designers are able to specify customized user characteristics instead of standard characteristics that are difficult to interpret because of their general nature. Besides, it provides designers the ability to characterize different groups of individuals that define the same characteristics, and so the user characterization can be easily reused.

This separation also allows the definition of static and dynamic characteris-
tics at the same time in the same space of definition.

Finally, another advantage of using the UsiXML methodology is the separa-
tion of the definition of concepts and tasks from the definition of UIs. Thus, the
characterization of users can be modified without having to modify the abstract user interface model, and vice versa.

As future work, we are actually working in the definition of an extension of the user awareness in order to model the social awareness of the user interfaces. The social awareness allows the UI to be aware not only of the user is operating it; it makes the UI be aware of other users that are part of the system. Thus, we cover the description of the social environment of context aware applications.

Another issue we have considered as part of future works is the inclusion of the location awareness as part of the UI specification to cover other context aware characteristics of the UI, such as the infrastructure environment, user position, etc.

Finally, we are also working on the definition of a common feature-based framework allowing designers to express characteristics that are related to the combination of the social and location features of context-aware UIs, such as the co-location.

## References

1. Barclay, P.J., Griffiths, T., McKirdy, J., Kennedy, J.B., Cooper, R., Paton, N.W., Gray, P.D.: Teallach - a flexible user-interface development environment for object database applications. J. Vis. Lang. Comput 14(1), 47-77 (2003)

2. Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interacting with Computers 15(3), 289-308 (2003)

3. Crease, M., Gray, P.D., Brewster, S.A.: A toolkit of mechanism and context independent widgets. In: Proceedings of DSV-IS. pp. 121-133 (2000)

4. Dey, A.K.: Understanding and using context. Personal and Ubiquitous Computing 5, 4-7 (2001)

5. Eisenstein, J., Vanderdonckt, J., Puerta, A.R.: Applying model-based techniques to the development of uis for mobile computers. In: IUI. pp. 69-76 (2001)

6. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading/MA (1995)

7. Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., López-Jaquero, V.: USIXML: A language supporting multi-path development of user interfaces. In: Bastide, R., Palanque, P.A., Roth, J. (eds.) EHCI/DS-VIS. LNCS, vol. 3425, pp. 200-220. Springer (2004)

8. Luyten, K., Laerhoven, T.V., Coninx, K., Reeth, F.V.: Runtime transformations for modal independent user interface migration. Interacting with Computers 15(3) (2003)

9. Mori, G., Paternò, F., Santoro, C.: CTTE: Support for developing and analyzing task models for interactive system design. IEEE Trans. on Soft. Eng. 28, 797-813 (2002)

10. Paternò, F., Santoro, C., Mäntyjärvi, J., Mori, G., Sansone, S.: Authoring pervasive multimodal user interfaces. Int. J. Web Eng. Technology 4(2) (2008)

11. Schmidt, A., Beigl, M., Gellersen, H.W.: There is more to context than location. Computers & Graphics 23(6), 893-901 (1999)
12. Spriestersbach, A., Ziegert, T., Grassel, G., Wasmund, M., and, G.D.: A single source authoring language to enhance the access from mobile devices to web enterprise applications. In: WWW2003 Developers Day Mobile Web Track. p. (not printed). Springer Verlag, Heidelberg etc., Germany (2003)

13. Tesoriero, R.: CAUCE: Model-driven Development of context-aware applications for ubiquitous computing environments. Ph.D. thesis, University of Castilla-La Mancha (December 2009)

14. Vanderdonckt, J.: A MDA-compliant environment for developing user interfaces of information systems. In: Proc. of 17 th Conf. on Advanced Information Systems Engineering CAiSE’05. pp. 13–17. Springer-Verlag (2005)

15. Vanderdonckt, J.M., Bodart, F.: Encapsulating knowledge for intelligent automatic interaction objects selection. In: Ashlund, S., Mullet, K., Henderson, A., Hollnagel, E., White, T. (eds.) Proc. of the Conf. on Human Factors in computing systems. pp. 424–429. ACM Press, New York (April 1993)

16. Weiser, M., Brown, J.S.: The coming age of calm technology. In: Denning, P.J., Metcalfe, R.M. (eds.) Beyond Calculation: The Next Fifty Years of Computing, pp. 75–85. Copernicus (1997)
# The Secret Lives of Assumptions: Developing and Refining Assumption Personas for Secure System Design

Shamal Faily and Ivan Fléchais

Oxford University Computing Laboratory
Wolfson Building, Parks Road, Oxford OX1 3QD UK
{shamal.faily,ivan.flechais}@comlab.ox.ac.uk

**Abstract.** Personas are useful for obtaining an empirically grounded understanding of a secure system's user population, its contexts of use, and possible vulnerabilities and threats endangering it. Often, however, personas need to be partly derived from assumptions; these may be embedded in a variety of different representations. Assumption Personas have been proposed as boundary objects for articulating assumptions about a user population, but no methods or tools currently exist for developing and refining these within the context of secure and usable design. This paper presents an approach for developing and refining assumption personas before and during the design of secure systems. We present a model for structuring the contribution of assumptions to assumption personas, together with a process for developing assumption personas founded on this model. We also present some preliminary results based on an application of this approach in a recent case study.

## 1 Introduction

Personas are useful for obtaining a grounded understanding of a system's contexts of use, and communicating that understanding within a design team. Recent work on applying personas to help elicit and specify secure system requirements found that the data and analysis from which personas are derived also help identify threats and vulnerabilities [10]. Although adherents of personas argue that these should be primarily derived from real-world observations [7, 14], the necessary resources for eliciting and analysing such data may not always be available. In these cases, it is necessary to rely on second-hand data about users and their contexts, much of which might be derived from assumptions.

Many usability professionals are familiar with analysing assumption-based usage data, but this may not be the case for software engineers. Engineers are usually employed for their technical expertise and domain knowledge; we cannot reasonably expect them to have a working knowledge of usability design techniques as well. They do, however, have tacit knowledge of the problem domain and a sensitivity to the values at play within its contexts of use. The challenge is to not only trace assumptions made about personas to their source, but to
explicate the claims these assumptions represent. By doing so, we also explicate
tacit knowledge about users and their contexts. Like data directly elicited from
real-world observations, this data also suggests hitherto unknown threats and
vulnerabilities related to a system.

Techniques from Design Rationale research are useful for tracking the refine-
ment of assumptions to architectural components and software. Such techniques
may also be useful for tracking the same assumptions to less refined concepts
used in security analysis. Security design has the same needs for discharging
potential ambiguity grounded in assumptions; these may be sources of attack
vectors if the vulnerabilities they expose are exploited. In this paper, we present
an approach for developing assumption personas for secure system design, and
describe how this approach can be embedded into an existing design process and
associated tool-support. In section 2, we briefly introduce personas and describe
the related work motivating our approach. In section 3 we present an overview
of our approach, and in section 4 we report on some preliminary findings which
arose when applying this approach in a recent case-study.

2 Related work

2.1 Personas and Assumption Personas

Personas are behavioural specifications of archetypical users. These were intro-
duced by Cooper [6] to deal with programmer biases arising from the word user.
These biases lead to programmers bending and stretching assumptions about
users to meet their own expectations; Cooper called this phenomena designing
for the elastic user. Personas are now a mainstay in User-Centered Design, with
articles, book-chapters, and even a book [14] devoted to developing and applying
them in practice. Personas have also been applied to Requirements Engineering,
an area of intersection between HCI and Software Engineering [4].

Accepting that data-driven personas are an ideal rather than a norm, Pruitt
& Adlin [14] proposed *Assumption Personas*: persona sketches created to ar-
ticulate existing assumptions about an organisation's user population. These
personas are grounded in assumptions contributors hold about users, and the
context of investigation. These assumptions may be derived from interpreted
or mis-interpreted experiences, and coloured by individual and organisational
values. Assumption Personas help people see the value of personas in design,
and how different assumptions shape them. As a result, when exposed, they can
guide subsequent analysis or data collection for data-driven personas.

Personas are not, however, without their critics. Chapman & Milham argue
that, as fictional archetypes, personas are difficult to verify as there is no way to
falsify them [5]. They further argue that questions remain about how personas
should be reconciled with other information, understanding what data underpins
their characteristics, and what happens when different interpretations are made
from the same persona.
## 2.2 Integrating Personas with Secure Software Engineering

Chapman and Milham's criticism about the stand-alone nature of personas can be addressed by integrating them into the software engineering process. This has been the subject of our recent work on the IRIS (Integrating Requirements and Information Security) framework, which integrates usability into the design of secure software systems [8]. As part of this work, a meta-model for usable secure requirements engineering was devised, which integrates the persona with other concepts in usability, security, and software engineering. From this model, we have developed CAIRIS (Computer Aided Integration of Requirements and Information Security): a tool for managing information about personas and other design elements, and evaluating the effect to security and usability of different design decisions [1]. CAIRIS manages requirements, task, and risk data, and automatically generates different types of visual model to represent the ongoing analysis. We demonstrate this approach in [9] by illustrating how categorical information about a task performed by a pre-defined persona is associated with the results of risk analysis, and how the usability of this task can be visually represented before and after a related risk is mitigated.

In [10], we presented a process for developing personas for secure systems; this is based on collecting and analysing empirical data from qualitative and contextual interviews. The personas derived from analysing this data were validated and further refined in participatory requirements and risk analysis workshops. We also found that empirical data used to derive personas could be re-used for other analysis.

Even though personas may be grounded in empirical data, the quandary about the validity of personas remains. It may be possible to verify the quality of the empirical-data or the robustness of the methodology to develop them, but we cannot easily falsify the representativeness of personas. The vision of the system may be tentative enough that what may have been valid working assumptions at the beginning of the persona development process may be invalid by the time the personas are presented to project stakeholders. It is, therefore, useful to understand how characteristics about personas track back to their assumptions, and why.

## 2.3 Toulmin's Model of Argumentation

Codifying the rationale underpinning assumption personas guides analysis and decision making, but the *rationale capture problem*, characterised by the reluctance of those involved in design activities to record their rationale, cannot be ignored [3]. Although the Design Rationale community has proposed several different approaches for building rationale capture into the design process, the Security and Requirements Engineering community has taken a recent interest in capturing rationale using the vehicle of informal argumentation. These approaches are founded on Toulmin's Argumentation Model: a logical structure for reasoning about the validity of arguments [15], the elements of which are defined in table 1.
<table><caption>**Table 1.** Elements of Toulmin's Argumentation Model</caption><thead><tr><td>Element</td><td>Description</td></tr></thead><tbody><tr><td>Claim</td><td>A proposition representing a claim being made in an argument.</td></tr><tr><td>Grounds</td><td>One or more propositions acting as evidence justifying the Claim.</td></tr><tr><td>Warrant</td><td>One or more rules of inference describing how the Grounds contribute to the Claim.</td></tr><tr><td>Backing</td><td>The knowledge establishing the Grounds for believing the Warrant.</td></tr><tr><td>Modal Qualifier</td><td>A phrase qualifying the degree of certainty in the argument for the Claim.</td></tr><tr><td>Rebuttal</td><td>One or more propositions challenging the validity of the Claim.</td></tr></tbody></table>

Alexander & Beus-Dukic describe a number of simple rationale models for Requirements Engineering based on this structure [2]. From a security standpoint, Haley et al. have proposed using Toulmin's model to support arguments for security requirements [11]. In their approach, an argument for a system satisfying its security requirements is presented for analysis. Each proposition within this argument is treated as a Claim, and argued accordingly. Rebuttals represent *Trust Assumptions*; these can be countered as part of another security argument, or examined in subsequent threat modelling activities.

# 3 Approach

We have chosen to embrace, rather than ignore, the contribution assumptions make to assumption persona design. We propose a novel approach to structuring the contribution of assumptions to persona specifications, and integrating this conceptual structure into an existing approach for secure systems design.

## 3.1 Developing assumption personas

Personas are usually represented as a narrative describing the behaviour of an archetypical user. Authoring these narratives remains a creative exercise, but we propose augmenting these by structuring the assumption data contributing to them. We have aligned this structure to Toulmin's Model of Argumentation, introduced in section 2.3. Adopting this approach allows us to treat assumptions directly contributing to part of the narrative as a Claim. The task of justifying this Claim both strengthens the foundations of the persona, and guides subsequent elicitation and analysis activities. These Claims are represented conceptually using one of more *Characteristics*; these are propositions about a specific aspect of a persona's behaviour. Characteristics are categorised according to one of the behavioural variable types defined by IRIS personas; these are based on the behavioural variable types proposed by Cooper [7]: activities, attitudes, aptitudes, motivations, and skills. Also associated with a Characteristic is a
<figure><img src="image_56.png" /><figcaption>**Fig. 1.** Conceptual model of assumption persona data (left), and Toulin model visualisation based on an individual characteristic (right)</figcaption></figure>

qualifying phrase representing the strength of belief in the Characteristic; this
qualifying phrase aligns with the Modal Qualifier in Toulmin's model.

Persona Characteristics originate from one of two sources. The first source is some form of *Artifact*: a document related to the problem domain or the system being specified, such as a specification, or a transcript from an interview or design workshop. The second is a design *Concept*: an instance of an object defined within the work-in-progress IRIS analysis, such as a description for an asset, a goal or requirement, or even another persona. Because an individual source may give rise to multiple Characteristics of the same or different behavioural categories, a *Reference* is associated with a given source and Characteristic. The contents of a Reference will depend on the source type. In the case of an Artifact, a reference contains information tying an attributable piece of information or comment to a source document or verbal comment, e.g. page number, document version, or person. In the case of a Concept, a Reference contains the name and type of the contributing concept. In both cases, the Reference will contain as much textual attribution information as necessary to justify the persona's Characteristic. The name of the Reference object is a synoptic proposition of this attribution information. With regards to Toulmin's model, References align to either Grounds, Warrants, or Rebuttals. Where a Reference represents a Warrant, the corresponding Artifact or Concept acts as the Warrant's Backing.

The meta-model in figure 1 (left) summarises these concepts and their re-
lationships. The stereotypes adjacent to each class represent the corresponding
concept name from Toulmin's model.
## 3.2 Applying and refining the assumption personas

Before assumption personas are used, they are presented to a workshop or focus group containing representative system stakeholders. Following this workshop, the remaining steps of the process are carried out in the context of smaller design sessions, as described by [10]. These sessions entail requirements and risk analysis activities, where, rather than referring to users, personas are used in their place. In both the workshop and design sessions, new assumptions about personas may be identified, or existing assumptions challenged. Armed with the proposed meta-model, tool-support can be developed to elicit the structural elements of the assumption persona argumentation model. Aside from guiding and structuring the elicitation of assumption data, the structured argument of Characteristics can be cross-checked with the persona narrative. If it becomes difficult to write a believable narrative based on the Characteristics identified, then these need to be re-evaluated.

We modified the CAIRIS tool introduced in section 2.2 to illustrate how tool-support can take advantage of this approach. As well as allowing Characteristics associated with a persona to be quickly reviewed against the narrative, we found that Characteristics could be quickly created or modified when assumptions are introduced or challenged during design sessions. Structuring the data according to the meta-model also facilitates the automatic generation of visual Toulmin models for persona Characteristics. An example of such a model for a specific Characteristic is provided in figure 1 (right).

Unsubstantiated Claims and Rebuttals are also an additional source of risk analysis information. In the case of the latter, obstacles – conditions representing undesired behaviour preventing an associated goal from being achieved [12] – can be elicited from these, and its placement guided by the related Characteristic negated by the Rebuttal. This placement guidance is possible because a persona invariably participates in tasks operationalised by one or more goals or requirements.

# 4 Preliminary Results

We used this approach to help specify requirements for an online portal for a medical research project. The nature of this project was such that eliciting empirical data from representative users during the study was impossible. During the course of the project, an assumption persona – Alex – was developed to embody the assumptions held by the project team about the researchers expected to use the portal. The assumptions underpinning this persona were initially derived from a high-level requirements specification document developed by a different team within the same project; as such, Alex represented the assumptions that team had made about the expected user population. After developing this persona, a half-day workshop was held with the complete project team to agree the scope for a subsequent requirements and risk analysis of the portal. During this workshop, Alex was presented to the team. The team both agreed and disagreed with the characteristics of Alex. Where there was disagreement,
the structured nature of the assumption data was used to track the questionable characteristic to its originating source, which was discussed in more detail within the team. Following the workshop, a number of new assumptions were elicited, which formed the basis of new characteristics about Alex.

After the workshop, three 2-hour design sessions were held with team members to carry out requirements and risk analysis relating to two specific tasks carried out by Alex. As part of this analysis, scenarios were developed describing how Alex would carry out these tasks with the aid of the portal. During these sessions, Alex's characteristics evolved; by the end of the 3rd session, 23 different Characteristics about Alex had been captured. Some of these were modifications to assumptions captured in the initial stages of persona development, but several were derived from assumptions which surfaced while eliciting other concepts, such as tasks and goals. In all cases, these characteristics were justified by Grounds, and in many cases, a Warrant and Backing were also elicited.

Haley & Nuseibeh [13] observed that experts provide essential domain knowledge about the subtleties of threats, but non-experts ask journalist questions challenging implicit assumptions assumed by the domain expert. Our preliminary results during the design sessions concur with this observation. When the tasks carried out by one of the personas was modelled during one session, one non-expert participant raised pertinent points about implicit assumptions in the task description; these were not accounted for by the personas, and led to the rebuttal of one Characteristic.

Although identifying Grounds for Characteristics was found to be straightforward, identifying Warrants provided to be more difficult. In particular, we found that, prior to their initial validation, many of the Characteristics were based exclusively on Grounds, rather than Warrants as well. As such, value judgements about the source data and the context were directly reflected in these Characteristics. Although the initial workshop surfaced a number of these issues, it was usually not until the personas were directly used to model tasks in design sessions that many invalid Characteristics were identified. Applying the personas within a specific context did, however, help identify missing inferential data, or guide the refactoring of the argumentation structure for affected Characteristics.

# 5 Conclusion

Personas are a mainstay in User-Centered Design, yet there is a dearth of guidance on how to build and refine these from assumptions, as opposed to empirical data. We believe this guidance, and corresponding tool-support, may contribute to a wider adoption of personas in secure software engineering, and a better understanding of how to use these in a secure software engineering context. This paper makes three contributions towards these ends. First, we have presented a model for structuring the assumptions contributing to personas; to help guide subsequent analysis, this model has been aligned with Toulmin's Model of Argumentation. Second, we have illustrated how tool support reifies this structured model, and guides subsequent risk analysis. Finally, we have reported some
of the preliminary results validating our approach in a recent case study. A more
detailed report of this study will appear as a future publication.

# 6 Acknowledgements

The research described in this paper was funded by EPSRC CASE Studentship
R07437/CN001. We are very grateful to Qinetiq Ltd for their sponsorship of this
work.

References

1. CAIRIS web site. http://www.comlab.ox.ac.uk/cairis

2. Alexander, I., Beus-Dukic, L.: Discovering requirements: how to specify products and services. Wiley, Chichester, West Sussex, England (2009)

3. Burge, J.E., Carroll, J.M., McCall, R., Mistrik, I.: Rationale-Based Software Engineering. Springer (2008)

4. Castro, J., Acua, S., Juristo, N.: Integrating the personas technique into the requirements analysis activity. In: Computer Science, 2008. ENC ’08. Mexican International Conference on. pp. 104–112 (Oct 2008)

5. Chapman, C.N., Milham, R.P.: The persona’s new clothes: Methodological and practical arguments against a popular method. Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (2006)

6. Cooper, A.: The Inmates Are Running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity (2nd Edition). Pearson Higher Education (1999)

7. Cooper, A., Reimann, R., Cronin, D.: About Face 3: The Essentials of Interaction Design. Wiley (2007)

8. Faily, S., Fléchais, I.: A Meta-Model for Usable Secure Requirements Engineering. In: Software Engineering for Secure Systems, 2010. SESS ’10. ICSE Workshop on. pp. 126–135. IEEE Computer Society Press (May 2010)

9. Faily, S., Fléchais, I.: Analysing and Visualising Security and Usability in IRIS. In: Availability, Reliability and Security, 2010. ARES 10. Fifth International Conference on (2010)

10. Faily, S., Fléchais, I.: Barry is not the weakest link: Eliciting Secure System Requirements with Personas. In: BCS HCI ’10: Proceedings of the 2010 British Computer Society Conference on Human-Computer Interaction (2010), to Appear

11. Haley, C.B., Laney, R., Moffett, J.D., Nuseibeh, B.: Arguing satisfaction of security requirements. In: Mouratidis, H., Giorgini, P. (eds.) Integrating Security and Software Engineering, chap. 2, pp. 16–43. Idea Group (2007)

12. van Lamsweerde, A., Letier, E.: Handling obstacles in goal-oriented requirements engineering. Software Engineering, IEEE Transactions on 26(10), 978–1005 (2000)

13. Nuseibeh, B., Haley, C., Foster, C.: Securing the skies: In requirements we trust. Computer 42(9), 64–72 (Sept 2009)

14. Pruitt, J., Adlin, T.: The persona lifecycle: keeping people in mind throughout product design. Elsevier, Amsterdam (2006)

15. Toulmin, S.: The uses of argument. Cambridge University Press, updated edn. (2003)
# Dazed and Confused Considered Normal: An Approach to Create Interactive Systems for People with Dementia

Nasim Mahmud<fnref n="1" />, Joël Vogt<fnref n="2" />, Kris Luyten<fnref n="1" />, Karin Slegers<fnref n="3" />, Jan Van den Bergh<fnref n="1" />, and Karin Coninx<fnref n="1" />

**Abstract.** In Western society, the elderly represent a rapidly growing demographic group. For this group, dementia has become an important cause of dependencies on others and causes difficulties with independent living. Typical symptoms of the dementia syndrome are decreased location awareness and difficulties in situating ones activities in time, thus hindering long term plans and activities. We present our approach in creating an interactive system tailored for the needs of the early phases of the dementia syndrome. Given the increasing literacy with mobile technologies in this group, we propose an approach that exploits mobile technology in combination with the physical and social context to support prolonged independent living. Our system strengthens the involvement of caregivers through the patient's social network. We show that applications for people suffering from dementia can be created by explicitly taking into account context in the design process. Context dependencies that are defined in an early stage in the development process are propagated as part of the runtime behavior of the interactive system.

## 1 Introduction

In Europe, at least half of the elderly population who suffer from dementia, have mild form of dementia [24]. Elderly citizens who are suffering from any form of dementia, are increasingly dependent on their social environment. Although people in early stages of dementia, are often able to perform everyday tasks without help. Their dependence on caregivers and need for assistance increases as the dementia syndrome progresses. This dependency puts strains on both the patients and their caregivers. As a patient's dependency on caregivers grows,

<footnote n="1">¹ Hasselt University - tUL - IBBT, Expertise Centre for Digital Media, Wetenschapspark 2, B-3590 Diepenbeek, Belgium
{nasim.mahmud, kris.luyten, jan.vandenbergh@uhasselt.be, karin.coninx}@uhasselt.be</footnote>

<footnote n="2">² Department of Informatics, University of Fribourg, Boulevard de Pérolles 90, CH-1700 Fribourg, Switzerland
joel.vogt@unifr.ch</footnote>

<footnote n="3">³ Centre for User Experience Research (CUO), IBBT / K.U.Leuven Parkstraat 45 bus 3605, B-3000 Leuven, Belgium
karin.slegers@soc.kuleuven.be</footnote>
patients feel reluctant to leave their home for the fear of getting lost, losing track of time or forgetting their goals. While the person with dementia is outside on his own, caregivers are burdened by worrying for the patients' safety.

In sum, during the course of the dementia syndrome, patients are likely to be less autonomous. They increasingly depend on caregivers with respect to their everyday tasks and activities. Most of the cases, in the early or mild stages of dementia, the caregiver is a family member or friend [30], thus part of the informal social network of the patient. A solution to this problem might both increase patients independence and relieve caregivers from (parts of) the stress they experience while taking care of their relative with dementia. A potential solution might be found in the realm of virtual connectedness to the social context.

In this paper we show how to map two of the main issues with dementia on the design of a mobile interactive systems. Roughly, people with dementia suffer from being lost in time and lost in space. Space and time are two important aspects that are taken into account in typical context-aware user interfaces [10, 1]. Besides these typical contextual parameters, the social network itself is also considered as part of the context in our approach. This implies that these contextual parameters (time, space and social network) should be part of the design process for building an interactive systems for people with dementia.

In order to validate our concept, we have developed a prototype of an interactive system, which is an aiding tool for persons with dementia. The prototype was built taking into account the important factors for independence of persons with dementia as mentioned above time, location and social network of a person with dementia to provide context aware help. It offers help only when it is necessary, without patronizing the user all the time.

# 2 Background

## 2.1 Dementia

Dementia is a term for a syndrome related to the loss of cognitive functions. It is usually the result of conditions that cause dysfunction of the brain. Draper [12] defines the dementia syndrome as an acquired decline in memory and thinking (cognition) due to brain disease that results in significant impairment of personal, social or occupational function. He explains that other brain functions such as orientation, language, reasoning and language processing are also affected. Dementia progresses over time. However the progression of dementia is individual and different brain functions do not decline at the same rate, some symptoms develop later in the course of the disease than others. The progression of dementia can be roughly classified according to the degree of severity. For example, by applying the scale developed by Leonard Berg [20]. The focus of this paper is on questionable and mild dementia. Further stages require increasing personal caring, which is outside the scope of our work. Dementia also puts enormous strains on the patients social context, especially as dementia progresses [12]. In most cases, the caregivers are spouses and when they can no longer care for the
patient, their children take over [30]. In a comparative study between caregivers for dementia and caregivers for non-dementia people, Ory et al. [26] found that the former was in almost all aspects more demanding than the latter.

## 2.2 Assistive Social Navigation

As mentioned earlier a person with dementia needs help with memory, spatial and temporal orientation. A context aware reminder and navigation system can help to partly regain independence by providing focused navigation information when necessary.

People often make decisions about their actions based on what other people have done in the past or on what other people have recommended them. This process is referred to as everyday social navigation [11]. When people need clue about some actions or things, they mostly base their decisions and actions on recommendations by people they know or people who share at least some common ground (e.g. person buying a book may also want to see what other people bought, who bought the same book). The process of finding the required information source and identifying the relevant information requires a lot of effort when it has to be performed in situ. An alternative method is to gather information by asking other people for advice [21]. Identifying the appropriate person for answering questions in a particular context and in suitable time is not a straightforward task [19], it is difficult even for people who do not suffer from dementia.

This becomes more difficult for a person with dementia, especially when he is spatially or temporally disoriented. Because of the disorientation, he becomes frustrated and it gets more difficult for him to formulate his enquiry. Assistive social navigation is *tailor made* information provided as suggestion for a particular end user. The suggestion is produced by a familiar person e.g. caregiver. In our case, the end user is the elderly person with dementia who needs information or help to complete any tasks by himself. When a person with dementia is lost in time or space (e.g. forgets where he is, where he is going or what he is supposed to do), he usually cannot get back to his normal state immediately. A person with dementia is often aware that he is lost, or he forgot his goal [29]. In this kind of frustrating situation he is unable to carry out his normal activity. Often he can recall his goal with some clues. But only his informal caregiver (e.g. spouse), who is aware about him and his tasks, can easily provide that targeted clues. Thus the caregiver can assist the person with dementia to navigate to complete his tasks only when it is needed; maintaining his autonomy when he is not disoriented.

## 2.3 Technologies for Dementia Aid

The need for independence for both people with dementia and their caregivers is commonly agreed on by researchers in the field [31, 29, 14]. Mulvenna et al. [24] conducted a study to analyze the need of people with dementia and their caregivers. They chartered the needs for more independence by means of *memory*
assistance and by providing more context such as *spatial and temporal orientation, social contact and, social interaction*. The person(s) who are in early stages or in mild stages of dementia are capable of living independently, and have occasional need for help [2]. A context-aware system can exert such support without patronizing the user. Based on these findings, the context-of-use will be explicitly taken into account in our approach to create an *interactive systems* that prolog independent living of elderly people with early and mild dementia.

Over the last decade, numerous IT-based support system for elderly people with dementia have been proposed [22, 17, 29]. One of the main objectives of those was to ensure helping elderly people live a safe and independent life. Providing cognitive support as a way of improving quality of life in general [16] has also been proposed. There are different ways in which such system offered support for independent living in some extent such as, tele-monitoring, reminding important activities [13], such as time to take medication etc. Reminder is a form of usually short message that helps people remember what is to be done. It consists of two independent features, signal and description [28]. An alarm clock could be a signal only reminder whereas an email notification is both visual signal and text description of the task to be done. Dey et. al. [9] proposed context-aware reminder system, other reminder system taking into account time, location, user present in the vicinity, user's activity.

Robinson et al. [29] developed a prototype to facilitate independent living for patients with light dementia that includes a communication platform for patients with dementia and their families. Hettinga et al. [16] evaluated the safety of navigation aid for people with mild dementia. The authors conclude that there is no evidence suggesting that the use of navigation devices for people with mild dementia are unsafe. Miskelly used a tracking system to define safe zones for persons with dementia. When the dementia person left the safe zone, the caregiver would receive a notification with name of person, location and cause [23]. Social awareness has been studied to enrich context information to provide context aware and appropriate help for nomadic persons [19, 5], and limited social awareness such phone book with the picture of social contact for persons with light or mild dementia has been proposed [8]. In our approach, we combine navigation aid and tracking system along with reminder service and social awareness as a mediator, to provide a tool to plan and assist activities of daily living to regain independence.

## 3 Scenario

This section motivates the use of an assistive application for a person with dementia. We use the COMuICSer approach presented by Haesen et al. in [15]. This approach uses a storyboard to reveal important information on the context of the user and helps to elicit the requirements to be taken into account. Furthermore, based on [18] it provides us with context information for usage in other models instantaneously.
A storyboard illustrating a regular day for a person with mild dementia is shown in Fig. 1. The storyboard depicts the sequence of activities that the person with dementia performed during that day. At one point, the person with dementia loses his orientation and cannot perform his task without help. At this point, the storyboard introduces other roles, such as a caregiver, that support the target user of our system. This information will be fed in the construction of the social network underpinning the target application.

<figure><img src="image_57.png" /><figcaption>**Fig. 1.** A day in Mr. John Doe's life, a) Jane helps John to plan his day, b) John meets his friend at the restaurant, c) John buys groceries and d) John is lost</figcaption></figure>

The depicted scenario of Fig. 1 goes as follows: John is in his early seventies, married to Jane. They have been living in a small village for more than three decades. Their daughter Elsa is married and lives in the same village. John is a retired school teacher. He is an enthusiastic bird watcher and spends a lot of his time on this hobby. Since his retirement, he enjoys reading books and hiking. Lately he started forgetting things but managed to hide it from Jane by taking notes or blaming it on others. But now he started having difficulties to find his
way back home. This happens even when he is in familiar places he has known for years. Initially he took this very lightly and did not pay attention. Until one day it took John an exceptionally long time to arrive back home; he was lost and came back with the help of nomadic foresters. Jane became highly worried after this incident. She started watching her husband more closely and noticed that her husband forgets small things on a regular basis as well. After talking to the family doctor and friend, Dr. Miller, he was diagnosed as having early stage of dementia. This diagnosis was made two months ago. Since then, John has been quite sad. He often worries about his future and gets angry a lot when he cannot remember things.

Today, John planned to go to local restaurant, and to the grocery shop after-
wards. Next, he will come back home. In the late afternoon at 5PM he is going
to the see Dr. Miller. In the morning, Jane made a simple to-do list on paper.
She also wrote their home phone number, the phone number of their daughter
Ilsa and the phone number of Dr. Miller, in case she cannot be reached. She
wanted to make sure that John does not forget all his tasks that he wanted to
perform, and that he does not feel afraid of getting lost again. It was already
after 4PM and John did not come back home yet (shown in the last frame of
Fig. 1). Jane started worrying, should she call him to check everything is OK
and remind him where he needed to go?

Besides having the sequence of actions crystallized and overview of the con-
text of use by using the COMuICSer approach, it also provides support for
describing the personas involved in the scenario. Next in our approach, these
personas will be converted into roles in the user interface. The personas can
also be used as filters when integrating the social network, i.e. when the patient
requires support from a specific type of caregiver the persona can be used as
an indication for this type. In our scenario a doctor acts as the caregiver in the
depicted situation.

**3.1 Elicited Requirements**

The scenario presented in the previous section describes the current situation and reveals the requirements for an interactive system for persons with dementia. We want to emphasize the following context of use especially plays an important role for our target application:

- the dementia person's time and location

- the caregivers’ time and location

- the social context and the participants' states, e.g. a caregiver is busy or available

Time and location deserve special attention, since these are two parts of the
context of use that a person with dementia has the most difficulties with. This
means a mismatch between the time and location of the person with dementia
with the time and location on which a specific task should be executed needs
to be explicitly handled by the resulting interactive system. A task and dialog
model provide more insight in how the system can actually support the user. The scenario further hints at the computing devices and communication services that should be available to support an improved scenario. For our purpose, a smartphone is sufficient to support the tasks and activities at hand.

# 4 Approach

## 4.1 User Interface Design Considerations for People with Dementia

Based on the scenario outlined in the previous section and related work in Sect. 2.3, it is clear that the design of an interactive system for elderly people with symptoms of early or mild dementia poses a number of challenges. Besides typical physical age-related impairments, such as reduced eye sight, decreasing motor skills or hearing difficulties, dementia adds further requirements to the interface design. The deteriorating short term memory, spatial and temporal orientation and the increasing difficulties with performing complex tasks make traditional user interfaces very difficult to use for dementia patients. Newell et al. [25] make several recommendations for the design of information technology for people with cognitive impairments. These are classified based on the type of impairment:

- Mitigate memory impairment. The interface should be simple and limit the possibilities for error. Users should be able to recognize errors and correct them. The system can assist by providing feedback and asking, where appropriate, the user to confirm an action and offer sensible reminders and prompts. Additionally, the user interface can assist the user's memory by providing navigation in the interaction.

- Avoid cognitive overload. The interface should limit options and be simple. Whenever possible, the dialog flow should be linear and parallel tasks possibly avoided.

- Take into account individual characteristics of dementia. Systems for patients with cognitive impairments must be adaptable to the patient's personal conditions that change over time.

The first two guidelines are appropriate for every end user and are especially desirable for people with dementia. Due to their cognitive decline, persons with dementia have difficulties performing parallel tasks and recalling the flow of task even if they are familiar with the task.

Context-awareness is a key feature to achieve personalized, situation-aware adoption of services, which is in accordance with the guidelines described above. Context-aware interactive systems are able to adapt according to the situational context in which they are executed. The context includes characteristics of the user, the device, the environment or a combination of these. Context can be used as input in the design process (for the parts that are known beforehand) and processed during runtime usage of an interactive system [6]. The latter ensures adaptability of the system while it is being used.
## 4.2 A Model Driven Design Approach

As appropriate handling of context is crucial to our application, we explored ways to effectively take all the relevant context influences into account from the start of the development process. Task models, such as the ConcurTaskTree (CTT) [27], are established starting points in the development process of interactive systems, such as the reference framework proposed by [6]. We explored the usage of the Contextual ConcurTaskTrees [4]. This allowed us to effectively represent influences of context changes on the tasks. However, the high-level of abstraction and the restricted set of information proved too limited to effectively create the application.

We thus looked for a more concrete notation that helped in clarifying the overall picture. Robinson et al. [29] developed applications for people with dementia and suggest eliciting the users requirement into the design by using storyboards. This improves the involvement of all stakeholders in the elicitation process. A storyboard is also a useful tool to capture the context of use. It clearly helps to highlight the social context of the persons who are communicating.

<figure><img src="image_58.png" /><figcaption>**Fig. 2.** Relation between the models used to develop the system. On the top, part of the storyboard showing the actors and their context. In the middle, the dialog model and arrows pointing the involvement of context and at the bottom, application model arrows showing influence on the dialog and the presentation model</figcaption></figure>

Section 3.1 and 4.1 discussed the requirements for the interactive system for a person with dementia. A storyboard is drawn that described the person with
dementia and the caregivers using the system that fulfills the requirements in Sect. 3.1. The storyboard shows a sequence of scenes depicting the situations and context in which the interactive system needs to operate. A scene shows the person with dementia and his mobile device and other (secondary) users (e.g. caregivers). We also add the time and place of a scene. Each scene is also accompanied by a textual description that further specifies the details. We used the COMuICSer tool [15] to annotate important characteristics in the storyboard (e.g. actors, location, time) and textual scene description.

The storyboard depicts the caregivers and the location and time when the user performs an activity. For example, one scene shows the user visiting the doctor at a certain time and place. The last scene shows the person with dementia getting lost. This scene hints at an incident where the person with dementia may require help. The context information time and location are associated with events that occur when certain predefined context rules are triggered. For example, the user is not at a given location at a predefined time or the user has reached the location for this task on time. The specific rules that are triggered by this context information are discussed into detail in Sect. 5.5. We further used low fidelity prototyping by sketching the user interfaces that would be shown to the user in the different scenes of the storyboard.

Based on the specifications captured by the storyboard (top of Fig. 2), we defined a dialog model (middle of Fig. 2) that describes the behavior of the application. The sequence of scenes indicates the states in the dialog model and how they are connected. The visual nature of the COMuICSer storyboard allows to easily identify all aspects of the context that are relevant for each state in the dialog model. Context events that are related to time and location trigger affect the flow of the dialog. The dotted arrows in Fig. 2 indicate this influence. In contrast with typical behavior of context-aware systems, the system does not adapt its presentation according to the context of use but tries to figure out how it can help the user to accomplish the tasks at hand. This means the context is not used to automate the system but rather the system works in a mixed initiative approach, the goal is to empower the user by pro-actively providing information cues and making suggestions.

Next, we need to create the presentation that exposes the behavior described in the previous paragraph. First, we have the presentation based on the sketches done in the earlier stage of the development. The presentation model and dialog model link is established by sketching the interfaces for the different scenes and clearly relating them with each other. There is also a link between the social context and the presentation model that needs to be taken into account. The availability states of caregivers are represented differently; we chose distinct colors for that purpose.

In the last step, the components of the system and their functionality are defined. This is taken as well from the requirements captured at the start of the development process. We identified a component for the to-do list manager, the social network, a component for location-based events and for time-based events. These components are described with the application model (bottom of Fig. 2).
<figure><img src="image_59.png" /><figcaption>**Fig. 3.** The development process, fine grained steps in which context, and context influences are explicitly described have borders with full lines</figcaption></figure>

The application model is then linked to the dialog and presentation model for orchestrating the behavior of the overall system.

The links between the application model and the dialog model are bi-directional: from the dialog model, functions described in the application model can be called. Dialog changes that are initiated by the application logic are triggered through time and location related events. Section 5.5 provides more details about these events. The link between the application model and presentation model determines how objects that are returned by an application are displayed. For example, depending on the availability status, a caregiver is displayed differently.

Figure 3 summarizes the different steps taken to develop our application. It clearly shows the mix informal and abstract formal notations during the development process and the explicit definition of context (influences) in almost all steps from the start of the development process. The task model step has a dashed border because the information it provided was not complete enough and could be completely replaced by the storyboard.

# 5 The Resulting System

## 5.1 Overview

Our system consists of three main building blocks: the user interface, context rules and the application logic. The user interface layer consists of three parts, namely 1) the to-do list and navigation module that is used by the person with dementia, 2) The to-do list manager that is the interface for the primary caregiver to manage the to-do list of the person with dementia and 3) the communication module that connects the person with dementia with the caregivers. The application logic consists of 1) the context module that receives time and location information, 2) the to-do list repository that stores the plans for the person with dementia and 3) the social network that connects the person with dementia with his or her social context. Between the user interface and the application logic reside the context rules. The rules use the current time and location, and assigned time and location for the current task to change the dialog.
<figure><img src="image_60.png" /><figcaption>Fig. 4. System overview</figcaption></figure>

The system runs on different communicating devices: the person with dementia uses a smart phone that runs Google Android. The communicating caregivers can use any Java enabled communicating devices that can connect to the phone network and the Internet. In the prototype described in this paper, the behavior of the system is presented in Java. The description of the user interface presentation uses the native XML user interface description language of the Google Android platform.

## 5.2 To-Do List and Navigation

The to-do list and navigation (see Fig. 4(a)) is used by the person with dementia to navigate through his plan. It receives the plan from the to-do list repository. The main navigation display shows information about the current task that is relevant to the user. This is an arrow indicating the direction where the user is expected to go, the name and time of the current task and a button that takes switches to the communication interface. An example of the to-do list and navigation interface is shown in Fig. 5(b). The to-do list and navigation module receives the location information from the context module through the context rules. This corresponds to the linking between the dialog model and the application model (see Fig. 2). The rules initiate intra and inter-dialog changes in response to the current time and location, and time and location associated with the current task.

## 5.3 To-Do List Manager

The to-do list manager (see Fig. 4(b)) is used by the primary caregiver, in most cases the spouse, to compile a plan for person with dementia. A plan is a set
of tasks that is uploaded to the to-do list repository which is accessed by the
to-do list navigator of the person with dementia. For each new task the caregiver
defines the name and description of the task, the time and location associated
with this task and possible relevant caregivers (e.g. daughter, brother, personal
physician). To avoid confusing the user, the list of the caregivers defined at this
stage are always displayed in the communication screen (see 5(c)).

5.4 Communication Module

The communication module (see Fig. 4(c)) offers a communication channel be-
tween the person with dementia and his caregivers. The person with dementia
can choose between selecting predefined text messages or making a phone call.
The list of available caregivers is provided during run-time via an XML-RPC
connection from the social network module, discussed in Sect. 5.7. The care-
givers are marked in a color that depicts their availability status. This is shown
in Fig. 5(c). It also contains an option to return to the navigation screen.

<figure><img src="image_61.png" /><figcaption>**Fig. 5.** The system in action, 5(a) visualization of the social network, dashed line rep-
resents unavailable user 5(b) navigation screen showing the navigation arrow pointing
towards the location of the current task to perform, as well as information about the
task and 5(c) communication screen showing list of caregivers, their availability status
is distinguished by colors</figcaption></figure>

5.5 Context Module and Context Rules

The context module (see Fig. 4(e)) listens for time and location information.
These are passed to the context rules (see Fig. 4(d)). The context module is
implemented as a location listener that is registered with the Google Android platform Location Manager to receive location updates. It further contains a local thread that queries the current time in predefined intervals. The context rules are executed after each time interval.

Context rules mediate between the context events sent by the context module, the current task and the dialog model. Based on time and location, intra or inter-dialog changes can occur.

- Intra-dialog changes occur for example when the user has reached the location for the task but is early or when the user is on time. In this case, the person with dementia is notified with an alert that pops up on the screen.

- Inter-dialog changes are triggered when a disorientation of the user is suspected. This is measured when the user has not reached the location for the task at the expected time. In this case, the user interface is changed to the communication screen. The user can at any time switch back to the to-do list and navigation screen.

## 5.6 To-do List Repository

The to-do list repository is a database that stores the lists of tasks that the person with dementia wants to perform. This database is accessed by the primary caregiver to create the to-do list (i.e. the plan). The rest of the time, it is accessed by the to-do list and navigation module to inform the user about the current task.

## 5.7 Social Network Module

The social networking module exploits the Ubiquitous Help System (UHS) [19] as a basis for the social networking communication platform. The UHS is based on a client-server architecture. An HTTP/XML based communication framework is used to facilitate a UHS client to communicate with other UHS clients. The UHS client can send and receive plain text and attached files such as a regular email. When a user needs help and asks query, the controller of the UHS sends the query to other clients via server. The controller of the receiving client initiates processing for profile and availability status matching, and replies with the appropriate information.

In the prototype we developed, a user is identified by a Friend-Of-A-Friend (FOAF) profile. This contains information about the social relations of the user and traditional information that identifies the user (such as name, address). FOAF is a Resource Description Framework vocabulary (RDF) [3] for describing people and social networks.

# 6 Discussion

In this paper we presented an interactive system and showed that applications for people with dementia can be created by explicitly taking context into account in
the design process. In our approach, context dependencies are defined in an early
stage in the development process. We utilized the COMuICSer storyboarding
tool to elicit our design requirements. This tool supports annotation of context
information (time, location, social context) and proofed to be more informative
than task models.

Typical context aware user interfaces consider space and time as important
aspects. In addition, our approach includes the social context of the users in-
volved in a communication process. The inclusion of social factors goes further
than simply using internal context of the user. We have exploited the Ubiquitous
Help System (UHS), introduced in earlier work, to empower social networking
feature into our prototype. This completes the different types of context that
need to be supported for applications that target people with early-stage de-
mentia.

However, in our current prototype, the interaction from the caregivers’ per-
spective has not been fully explored yet. Our aim was mainly to present an ap-
propriate approach that addresses the three different types of context (location,
time, social) that are important for these users explicitly during the development
of such a system. In our future work, this issue will be addressed in order to have
a system, which can be tested with real users.

The development of this prototype learned us that informal notations play an
important role in the development of context-sensitive interactive systems, such
as the one described in this paper. The explicit relations between the informal
description and the formal models, such as the dialog model, inspire us to explore
potential automation or consistency checks between the informal specifications
offered by the scenario model and the presentation sketches, especially since
there are already existing formalizations of both notations [18, 7].

Acknowledgments

The authors are pleased to thank Mieke Haesen of EDM for her help in utilizing
the storyboard <fnref n="4" /> tool. Part of the research at EDM is funded by EFRO (Euro-
pean Fund for Regional Development) and the Flemish Government. Funding for
this research is also provided by the Research Foundation - Flanders (F.W.O.
Vlaanderen, project CoLaSUE, number G.0439.08N) and one of the authors is
funded by the Hasler Foundation (LoCa project).

References

1. Balme, L., Demeure, A., Barralon, N., Coutaz, J., Calvary, G.: CAMELEON-RT: A software Architecture Reference Model for Distributed, Migratable, and Plastic user Interfaces. Ambient Intelligence 3295, 291–302 (2004)

<footnote n="4">⁴ StoryBoardML: http://research.edm.uhasselt.be/~kris/research/projects/StoryBoardML/storyboards.html</footnote>
2. Baumgarten, M., Mulvenna, M.: The Role of Context-aware Computing in Support of People with Dementia. In: Supporting People with Dementia Using Pervasive Health Technologies, pp. 131–143. Springer (2010)

3. Beckett, D., McBride, B.: RDF/XML Syntax Specification (Revised). Tech. rep., W3C (February 2004), http://www.w3.org/TR/rdf-syntax-grammar/, http://www.w3.org/TR/rdf-syntax-grammar/

4. den Bergh, J.V., Coninx, K.: Contextual concurtasktrees: Integrating dynamic contexts in task based design. In: PerCom Workshops. pp. 13–17. IEEE Computer Society (2004)

5. Bilandzic, M., Foth, M., De Luca, A.: CityFlocks: designing social navigation for urban mobile information systems. In: DIS ’08: Proceedings of the 7th ACM conference on Designing interactive systems. pp. 174–183. ACM, New York, NY, USA (2008)

6. Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interacting with Computers 15(3), 289–308 (2003)

7. Coyette, A., Kieffer, S., Vanderdonckt, J.: Multi-fidelity Prototyping of User Interfaces. In: Baranauskas, M.C.C., Palanque, P.A., Abascal, J., Barbosa, S.D.J. (eds.) INTERACT (1). Lecture Notes in Computer Science, vol. 4662, pp. 150–164. Springer (2007)

8. Davis, R., Nugent, C.D., Donnelly, M.: Prototyping Cognitive Prosthetics for People with Dementia. In: Mulvenna, M. D. & Nugent, C. D. (ed.) Supporting People with Dementia Using Pervasive Health Technologies, pp. 145–163. Springer (2010)

9. Dey, A.K., Abowd, G.D.: CybreMinder: A Context-Aware System for Supporting Reminders. In: HUC ’00: Proceedings of the 2nd international symposium on Handheld and Ubiquitous Computing. pp. 172–186. Springer-Verlag, London, UK (2000)

10. Dey, A.K., Häkkilä, J.: Context-Awareness and Mobile Devices. User interface design and evaluation for mobile technology 1, 205217 (2008)

11. Dourish, P., Matthew, C.: Running out of space: Models of information navigation. presented at HCI’94, Cambridge, UK NA, 134 (94)

12. Draper, B.: Dealing with dementia : a guide to Alzheimer’s disease and other dementias / Brian Draper. Allen & Unwin, Crows Nest, N.S.W. (2004)

13. Du, K., Zhang, D., Zhou, X., Mokhtari, M., Hariz, M., Qin, W.: HYCARE: A Hybrid Context-Aware Reminding Framework for Elders with Mild Dementia. In: ICOST ’08: Proceedings of the 6th international conference on Smart Homes and Health Telematics. pp. 9–17. Springer-Verlag, Berlin, Heidelberg (2008)

14. Graff, M., Vernooij-Dassen, M., Thijssen, M., Dekker, J., Hoefnagels, W., Rikkert, M.: Community based occupational therapy for patients with dementia and their care givers: randomised controlled trial. British Medical Journal 333(7580), 1196 (2006)

15. Haesen, M., Luyten, K., Coninx, K.: Get Your Requirements Straight: Storyboard- ing Revisited. In: INTERACT (2). pp. 546–549 (2009)

16. Hettinga, M., De Boer, J., Goldberg, E., Moelaert, F.: Navigation for People with Mild Dementia. Studies in health technology and informatics 150, 428 (2009)

17. Lauriks, S., Reinersmann, A., Roest, H., Meiland, F., Davies, R., Moelaert, F., Mulvenna, M., Nugent, C., Dröes, R.: Review of ICT-Based Services for Identified Unmet Needs in People with Dementia. In: Mulvenna, M. D. & Nugent, C. D. (ed.) Supporting People with Dementia Using Pervasive Health Technologies. Springer (2010)
18. Luyten, K., Haesen, M., Ostrowski, D., Coninx, K., Degrandsart, S., Demeyer, S.: Storyboard creation as an entrypoint for model-based interface development with UsiXML. In: UsiXML. pp. 1–8 (2010)

19. Mahmud, N., Luyten, K., Coninx, K.: Context Aware Help and Guidance for Large-Scale Public Spaces. In: SMAP ’09: Proceedings of the 2009 Fourth International Workshop on Semantic Media Adaptation and Personalization. pp. 105–110. IEEE Computer Society, Washington, DC, USA (2009)

20. Maj, M., Sartorius, N.: Dementia Volume 3 of WPA series, WPA series, vol. 3. John Wiley and Sons (2003)

21. McDonald, D.W., Ackerman, M.S.: Expertise recommender: A flexible recommen-dation system and architecture. In: Proceedings of the 2000 ACM conference on Computer supported cooperative work. pp. 231–240. ACM (2000)

22. Mileo, A., Merico, D., Bisiani, R.: Support for context-aware monitoring in home healthcare. J. Ambient Intell. Smart Environ. 2(1), 49–66 (2010)

23. Miskelly, F.: A novel system of electronic tagging in patients with dementia and wandering. Age and ageing 33(3), 304 (2004)

24. Mulvenna, M.D., Nugent, C.D., Moelaert, F., Craig, D., Dröes, R.: Supporting People with Dementia Using Pervasive Healthcare Technologies. In: Supporting People with Dementia Using Pervasive Health Technologies, pp. 3–14. Advanced Information and Knowledge Processing, Springer, London, UK (2010)

25. Newell, A., Carmichael, A., Gregor, P., Alm, N., Waller, A.: Information technol- ogy for cognitive support. In: Sears, A., Jacko, J.A. (eds.) The Human- Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Ap- plications, chap. 41, pp. 811–828. CRC Press, 2. edn. (2007)

26. Ory, M., Hoffman III, R., Yee, J., Tennstedt, S., Schulz, R.: Prevalence and impact of caregiving: A detailed comparison between dementia and nondementia care-givers. The Gerontologist 39(2), 177 (1999)

27. Paterno, F.: Model-Based Design and Evaluation of Interactive Applications (Ap- plied Computing). Springer (2000)

28. Ren, Y., Kiesler, S., Fussell, S.: Multiple group coordination in complex and dy-namic task environments: Interruptions, coping mechanisms, and technology rec- ommendations. J. Manage. Inf. Syst. 25(1), 105–130 (2008)

29. Robinson, L., Brittain, K., Lindsay, S., Jackson, D., Olivier, P.: Keeping In Touch Everyday (KITE) project: developing assistive technologies with people with de- mentia and their carers to promote independence. International Psychogeriatrics 21(03), 494–502 (2009)

30. Schulz, R., Martire, L.: Family caregiving of persons with dementia: prevalence, health effects, and support strategies. American Journal of Geriatric Psych 12(3), 240 (2004)

31. Woods, B.: Promoting well-being and independence for people with dementia. In- ternational journal of geriatric psychiatry 14(2), 97–105 (1999)
# Supporting Multimodality in Service-oriented Model-based Development Environments

Marco Manca, Fabio Paterno

CNR-ISTI, HIIS Laboratory, Via Moruzzi 1, 5614 Pisa, Italy
{Marco.Manca, Fabio.Patrono}@isti.cnr.it

**Abstract.** While multimodal interfaces are becoming more and more used and supported, their development is still difficult and there is a lack of authoring tools for this purpose. The goal of this work is to discuss how multimodality can be specified in model-based languages and apply such solution to the composition of graphical and vocal interactions. In particular, we show how to provide structured support that aims to identify the most suitable solutions for modelling multimodality at various detail levels. This is obtained using, amongst other techniques, the well-known CARE properties in the context of a model-based language able to support service-based applications and modern Web 2.0 interactions. The method is supported by an authoring environment, which provides some specific solutions that can be modified by the designers to better suit their specific needs, and is able to generate implementations of multimodal interfaces in Web environments. An example of modelling a multimodal application and the corresponding, automatically generated, user interfaces is reported as well.

**Keywords:** Multimodal interfaces. Model-based design. Authoring tools.

## 1 Introduction

Multimodal user interfaces support various user input modes. Ongoing technological evolution is making such interfaces more and more affordable and is proposing them in the mass market as well. However, developing multimodal user interfaces is still difficult and there is a lack of authoring environments for this purpose.

Model-based approaches have received renewed attention in recent years because they can help developers in managing the complexity of designing and developing multi-device applications. Most of the proposed model-based approaches have focused on desktop and mobile applications, sometimes with support for vocal interfaces as well, but there has been little effort in applying them to multimodal user interfaces, and such rare studies have found limited applications, as results were still too preliminary to provide general solutions.
In this paper, we present a logical language and an associated authoring environment able to provide a useful and general solution to such issues, and which can be exploited by developers of multimodal user interfaces. In the paper after discussing related work we introduce our approach to modelling multimodal interaction; next we show how it has been formalized in an XML logical language to address composition of vocal and graphical modalities, and we present how such language is supported within an authoring environment. Then, the transformation from the logical description to implementation is discussed, and an example multimodal application obtained through this environment is presented as well. Lastly, some conclusions are drawn along with indications for future work.

## 2 Related Work

The problem of designing multi-modal interfaces has been addressed in some previous work but still needs more general and better engineered solutions. Damask [7] includes the concept of layers to support the development of cross-device (desktop, smartphone, voice) user interfaces. Thus, the designers can specify user interface elements that should belong to all the user interface versions and elements that should be used only with one device type. However, this approach can be useful in developing single modality versions (graphical or vocal) but does not provide particularly useful support when considering multimodal interfaces, which require specific support to indicate how to compose the involved modalities. XFormsMM [5] is an attempt to extend XForms in order to derive both graphical and vocal interfaces. In this case the basic idea is to specify the abstract controls with XForms elements and then use aural and visual CSS for vocal and graphical rendering, respectively. The problem in this case is that aural CSS have limited possibilities in terms of vocal interaction and the solution proposed requires a specific ad hoc environment in order to work. For this purpose we propose a more general solution able to derive different implementations for desktop and mobile devices. Obrenovic et al. [11] have investigated the use of conceptual models expressed in UML in order to then derive graphical, form-based interfaces for desktop or mobile devices or vocal ones. UML is a software engineering standard mainly developed for designing the internal software of application functionalities. Thus, it seems unsuitable to capture the specific characteristics of user interfaces and their software. In [15] there is a proposal to derive multimodal user interfaces using attribute graph grammars, which have a well-defined semantics but limitations in terms of performance. The possibility of deriving mutlimodal interfaces was addressed in [12] but using hardcoded solutions for the transformation and logical descriptions that were unable to describe typical Web2.0 interactions and access to Web services.

A different approach to multimodal user interface development has been proposed in [6], which aims to provide a workbench for prototyping them using off-the-shelf heterogeneous components. In that case model-based descriptions are not used and it is necessary to have an available set of previously defined components
able to communicate through low-level interfaces, thus making it possible for a
graphical editor to easily compose them.

To summarise, we can say that the few research proposals that have also considered multimodal interfaces have not been able to obtain a general solution in terms of logical descriptions and provide limited support in terms of generation of the corresponding user interface implementations. For example, in [12] the transformations were hard-coded in the Java implementation, while in [15] the transformations were specified using attributed graph grammars, whose semantics is formally defined but have considerable performance limitations.

In this paper we present a general logical language for multimodal
interaction, which is included in an overall environment able to support development
of multi-device user interfaces. The associated authoring environment includes a
transformation tool able to derive X+V implementations from the logical
specifications and satisfies the requirements for multimodal interface generation
discussed in previous work [10], such as modality independence, support for
specifying hierarchical grouping, etc.

3 Background

MARIA [13] is a recent model-based language, which allows designers to specify abstract and concrete user interface languages according to the CAMELEON Reference framework [2]. This language represents a step forward in this area because it provides abstractions also for describing modern Web 2.0 dynamic user interfaces and Web service accesses. In its first version it provides an abstract language independent of the interaction modalities and concrete languages for graphical desktop and mobile platforms. In general, concrete languages are dependent on the typical interaction resources of the target platform but independent of the implementation languages. In this paper we present a concrete language for multimodal interfaces, which has been designed within the MARIA framework.

In MARIA an abstract user interface is composed of one or multiple
presentations, a data model, and a set of external functions. Each presentation
contains: a number of user interface elements (interactors) and interactor
compositions (indicating how to group or relate a set of interactors); a dialogue
model, describing the dynamic behaviour of such elements and connections,
indicating when a change of presentation should occur. The interactors are classified
in abstract terms, e.g. edit, selection, output, control. Each interactor can be associated
with a number of event handlers, which can change properties of other interactors or
activate external functions. While in graphical interfaces the concept of presentation
can be easily mapped on that of a set of user interface elements perceivable at a given
time (e.g. a page in the Web context), in the case of a vocal interface we consider a
presentation as a set of communications between the vocal device and the user that
can be considered as a logical unit, e.g. a dialogue supporting the collection of
information regarding a user. In defining the vocal concrete language [14] we have
refined the abstract vocabulary for this platform. This mainly means that we have
defined vocal refinements for the elements defined in the abstract language: interactors (user interface elements), the associated events and their compositions. The multimodal support has been built on top of such parts following an approach discussed in the next section.

# 4 Approach to Modelling Multimodal Interaction

In this paper we present a multimodal environment able to support composition of graphical and vocal interactions. There are many ways to compose such modalities. The goal is to provide a structured support that aims to identify the most suitable solutions at various granularity levels. In order to indicate how to combine the modalities, we have considered the well-known CARE properties (CARE: Complementarity, Assignment, Redundancy, Equivalence) [4] at various granularity levels. We apply such properties in the following manner:

* *Complementarity*: the considered part of the interface is partly supported by one modality and partly by another one;

* *Assignment*: the considered part of the interface is supported by one assigned modality;

* *Redundancy*: the considered part of the interface is supported by both modalities;

* *Equivalence*: the considered part of the interface is supported by either one modality or another.

How such properties will be applied to the user interface elements depends on the modalities and platforms considered. In the following, how these properties are applied to mixed vocal+graphical interfaces in both desktop and mobile devices is described, but the approach presented can be applied to other types of modalities. Since we want to provide a flexible environment, the possibility of applying such properties is supported in the definition of the various aspects characterising our logical descriptions: the composition operators, the interaction and the only-output elements. In addition, in order to have the possibility of controlling multimodality at a finer level, the interaction elements are structured into three phases (each of them can be associated with a different CARE property):

* *Prompt*: represents the interface output indicating that it is ready to receive an input.

* *Input*: represents how the user can actually provide the input.

* *Feedback*: represents the response of the system after the user input.

In practise, not all the CARE properties can be applied to all the three phases of an interaction. In particular, equivalence can be applied only to input: when two modalities are available and either one or the other can be used to enter the input. Vice versa, redundancy can be applied to prompt and feedback, but not to input, since a redundant input would mean that the same input is provided through different modalities, which does not seem useful or efficient. Complementarity could be applied to all three phases. However, in the case of input it can meaningfully be
applied when structured input are considered. Indeed, atomic inputs that require
simple actions (e.g. button selection) can hardly be obtained through a complementary
use of two modalities.

By default the tool provides some specific solutions in terms of possible CARE
properties, which can be modified by the designers to suit their specific needs. Figure
1 shows the control panel to define the CARE properties that are made available or
the refinement of the main abstract concepts (there is one tab for each of them). The
CARE properties that have been deemed not meaningful appear greyed out. Designers
can freely select those properties that seem more appropriate for their multimodal
applications, and then the authoring environment will be able to generate user
interfaces accordingly following transformations that will be introduced in the next
sections. Thus, our environment allows the designers to customize what multimodal
support to provide in user interface development.

<figure><img src="image_62.png" /><figcaption>Fig. 1. Control panel for customizing CARE properties.</figcaption></figure>

While the CARE properties made available are similar for the two types of platforms
that we consider (multimodal desktop and multimodal mobile), there are differences
in the default properties proposed by the environment, taking into account the richer
set of graphical resources of the desktop platform and that the mobile device can often
be used on the move. Thus, in the case of the multimodal desktop, which has rich
graphical resources, the composition operators are supported graphically. The
interaction elements are structured in such a way that the prompt is graphical, input
can be either graphical or vocal, and feedback is in both modalities. The only-output
elements are graphical. In the case of a multimodal mobile, which has less rich
graphical resources, the composition operators are supported both graphically and
vocally, and the interaction elements are supported in such a way that the prompt is
both vocal and graphical, the input either graphical or vocal, and the feedback is
expressed in both modalities. The only-output elements can be both graphical and
vocal or they use the two modalities in a complementary way, if they take a lot of
resources.
<table><caption>Table 2. How CARE Properties are made available for graphical+vocal desktop and mobile</caption><thead><tr><th>Element type</th><th>Interaction Phase</th><th>CARE Properties for Desktop</th><th>CARE properties for Mobile</th></tr></thead><tbody><tr><td>Composition Operator</td><td></td><td></td><td></td></tr><tr><td>Grouping Relation</td><td>Output</td><td>Graphical Assignment Redundancy</td><td>Vocal Assignment Graphical Assignment Redundancy</td></tr><tr><td>Only Output Interactor</td><td></td><td></td><td></td></tr><tr><td>Description, Object, Feedback, Alarm, Table</td><td>Output</td><td>Graphical Assignment Redundancy Complementarity</td><td>Vocal Assignment Graphical Assignment Redundancy Complementarity</td></tr><tr><td>Interaction Interactor</td><td></td><td></td><td></td></tr><tr><td rowspan="3">Single/multiple selection Text Edit Numerical Edit</td><td>Input</td><td>Graphical Assignment Equivalence</td><td>Graphical Assignment Equivalence Vocal Assignment</td></tr><tr><td>Prompt</td><td>Graphical Assignment Redundancy</td><td>Graphical Assignment Redundancy Vocal Assignment</td></tr><tr><td>Feedback</td><td>Graphical Assignment Redundancy</td><td>Graphical Assignment Redundancy Vocal Assignment</td></tr><tr><td rowspan="3">Activator</td><td>Input</td><td>Graphical Assignment Equivalence</td><td>Graphical Assignment Equivalence Vocal Assignment</td></tr><tr><td>Prompt</td><td>Graphical Assignment Redundancy</td><td>Graphical Assignment Redundancy Vocal Assignment</td></tr><tr><td>Feedback</td><td>Graphical Assignment Redundancy</td><td>Graphical Assignment Redundancy</td></tr><tr><td rowspan="3">Navigator</td><td>Input</td><td>Graphical Assignment Equivalence</td><td>Graphical Assignment Equivalence Vocal Assignment</td></tr><tr><td>Prompt</td><td>Graphical Assignment Redundancy</td><td>Graphical Assignment Redundancy Vocal Assignment</td></tr><tr><td>Feedback</td><td>Vocal Assignment None</td><td>Vocal Assignment None</td></tr></tbody></table>
Table 2 provides details on how the CARE properties are initially proposed by the environment to then generate graphical and vocal interfaces in both desktop and mobile platforms. Thus, it shows what properties have been deemed meaningful in the case of graphical and vocal interfaces, and these are made available in the authoring environment. We indicate in bold the specific properties that are initially pre-selected by default in the system. Thus, the properties in bold are those applied if the designer does not change anything in the tool. In particular, the first column indicates the element of the abstract interface considered. Different interaction phases (input, prompt, feedback) have to be considered depending on the interaction element in question.

In the case of only-output elements for the multimodal desktop platform the graphical assignment is proposed, while for the mobile one redundancy is suggested. For the interactive elements, in the desktop case we suggest equivalence for input and graphical assignment for prompt and feedback, while in the mobile case we prefer redundancy for prompt and feedback and still equivalence for input.

The composition operators aim to put together some interface elements in such a way that logical closeness or hierarchy of importance or some ordering is highlighted. Thus, usually there is some output information to indicate the involved elements (for example, it could be a graphical container or a sound at the beginning and the end of the grouped elements).

The navigator allows the user to move from one presentation of the application to another. This type of element usually has no immediate feedback because the actual feedback is given by the change of the application presentation loaded. However, it is possible to have some kind of vocal feedback to indicate that a change of presentation is under way.

# 5 A Logical Language for MultiModality

In the MARIA framework the concrete languages are derived from the abstract one by refining the abstract vocabulary taking into account the considered platform and the associated interaction modality. In the case of a multimodal concrete language we have to consider refinements for multiple modalities and indicate how to compose them. In particular, the MARIA concrete language for composing graphical and vocal modalities is based on the two previously defined concrete languages (one for the graphical [13] and one for the vocal modality [14]). It adds the possibility to specify how to compose them through the CARE properties.

As we introduced before the MARIA abstract language structures a user interface in terms of a number of presentation. Each presentation has composition operators (usually groupings). The composition elements contain interactors that can be either interaction or only-output interface basic components, which can have events handlers associated to them indicating how they react to events. Each of these components of the language, ranging from the presentations to the elementary interactors, have different refinements for the graphical and the vocal modality and in the multimodal concrete language we indicate how to compose them. Thus, a multimodal presentation has associated both graphical settings (such as background colour or image or font settings) and vocal settings (such as speech recogniser or
synthesis attributes). A grouping in the multimodal concrete language can exploit both visual aspects (using attributes such as position, dimension, border backgrounds) and vocal techniques (for example inserting keywords or sounds or pauses or changing synthesis properties). The interactors are enabled to exploit both graphical events (associated with mouse and keyboards) or vocal-specific events (such as no input or no match input or help request).

<figure><img src="image_63.png" /><figcaption>Fig. 2. An example of multimodal interactor derived from the graphical and vocal ones</figcaption></figure>
In order to better understand how this approach works, we can take an example abstract interactor, the text edit. At the abstract level there is no assumption regarding the modality that should be used to perform this interaction. In Figure 2 there is a graphical representation of how this abstract interactor is refined into two parts depending on the modality, and then there are the possible CARE properties that have deemed meaningful for this interactor (in the top part of Figure 2). In the graphical case we have either a text area or a text field interactor as possible refinement, while in the vocal case we obtain a vocal textual input, which is composed of a request, a grammar to specify possible inputs and the associated feedback. Thus, the multimodal language includes both the vocal and the graphical refinements of the interactor, and adds attributes associated with instances of the CARE properties, which indicate the possible ways to compose them in the various interaction phases (input, prompt, feedback).

# 6 The Transformation into an Implementation

In terms of target implementation languages, we have considered X+V [1] because it supports multimodality through the Web, which is the most common interaction environment, it is a standard and currently some publicly available browsers (such as Opera) support it, thus allowing developers to immediately test the resulting interfaces. X+V is an integration of HTML and VoiceXML. The VoiceXML part is included in the head of the X+V document, while the HTML is in the body part. Thus, there is a clear distinction between these two parts in an X+V implementation. The connection between the two parts is obtained through the events and the associated handlers. For example, the expression:

```
 <input type="text" id="from" name="departure city" ev: event="
    inputfocus" ev: handler="# voice city"/>
```

indicates that when the input focus event occurs in the from element of the graphical form then the voice.city event handler (which is managed in the vocal part) should be performed. In an X+V specification the synchronization between the values in the vocal and graphical part are obtained through the sync elements:

```
<xv: sync xv: input=" departure city " xv: field ="# departure city field "/>
```

This sync element associates the value of an input element in the HTML part (departure.city) with the indicated field VoiceXML element (departure.city_field). This means that when an element is entered vocally then it is associated with both the vocal field and the input HTML element. The same result is obtained if the element is entered graphically. In addition, if the user changes the focus in the graphical part,
then the corresponding vocal element, if any, is enabled. The sync element is not
located in the VoiceXML form but it is a direct child of the HEAD element.

User interface generation is obtained through XSLT transformations [3]. They are
obtained through stylesheets that transform an XML document into a new one in the
target language (in our case the XML languages involved are the multimodal concrete
MARIA language and X+V). The transformation is composed of a set template rules,
which are defined by patterns indicating the source nodes conditions that should be
verified, and templates indicating what corresponding element in the target document
should be included. For example:

<xsl:template match="cui:presentation">
<html>
<head><title>Presentation title</title></head>
<body>Presentation content</body>
</html>
</xsl:template>

Indicates that a presentation in the source concrete language should be associated with
the indicated elements in the corresponding HTML code.

The value of the CARE properties for the various user interface parts determines
what should be generated. Assignment indicates whether only the vocal or only the
graphical part is generated. Equivalence means that input in both modalities are
generated, in particular for the vocal part a VoiceXML field is generated, for the
HTML part an input element and then also a X+V element to synchronise the two
parts. Complementarity and redundancy require generation of both the graphical and
the vocal parts, even if they differ in the actual content that is generated.

The transformation is composed of three stylesheets: one for the graphical part and two for the vocal part, one to generate elements that are in already existing forms and one is for elements that require the creation of forms in which to put the currently generated element.

Thus, the transformation creates an X+V page for each presentation in the
concrete description in such a way that in the head tag there is the call of the template
to generate the X+V elements to synchronise the vocal and the graphical inputs and
the templates to generate the vocal elements, while in the body tag there are the
templates for generating the graphical elements. The X+V sync element is created
only for the implementation of those interactors that are associated with the
equivalence property for the input phase.

The transformation is also able to handle complex data structures such as tables.
In the case tables must be rendered vocally, then it is possible to support either linear
browsing (the elements are rendered line by line) or intelligent browsing, in which the
corresponding header is rendered for each data element as well.
# 7 Authoring an Example Application

Tool support for the method presented has been implemented and integrated in the MARIAE environment, which is publicly available at http://giove.isti.cnr.it/tools/Mariae/. In order to see how it works we can consider an example application. We consider a home application, which allows users to control a number of domestic appliances.

The application is composed of four presentations: one for the user login, one showing the rooms that it is possible to monitor, one showing the appliances in the room selected, and one to change the settings of the appliance selected, if any.

<figure><img src="image_64.png" /><figcaption>Fig. 3. Authoring a multimodal concrete presentation.</figcaption></figure>

Figure 3 shows the authoring environment in which the login presentation is being edited. The designer has specified a grouping element (*login_form*), which includes the input fields. It also contains a vocal element *grouping_start*, which is used to render a vocal message “Start login form!”. On the right-top part of the environment there is a panel for setting the multimodal attributes (the CARE properties) of the currently selected element. In the main central part there are the elements that compose the currently selected presentation. They are graphically represented as the XML syntax of the specification may be not easy to read and manage. The currently selected element highlighted in red is a text edit interactor for the entering of the user name. Since the CARE properties indicate the use of both
graphical and vocal modality it has a graphical part with a text edit interactor and a
vocal one with a vocal textual input interactor. The vocal part has two request
elements with the count attribute, which allows developers to implement the tapered
prompting technique. The first request asks for 'Insert your username'. In the case the
user does not provide an input within a given time or the input is not recognised then
the second request provides a more detailed indication of what has to be entered. The
vocal textual input also allows the specification of a grammar for which the grammar
options represent the possible inputs.

Figure 4 shows the multimodal implementation rendered through an Opera browser of
the login presentation.

<figure><img src="image_65.png" /><figcaption>Fig. 4. The multimodal user interface corresponding to the previous presentation</figcaption></figure>

Then, we can see (Figure 5) how it is possible to create connections among the
various presentations through the authoring environment. The interactor_id attribute
identifies the navigator interactor that triggers the presentation change, while
presentation name indicates the target presentation. The Figure also shows the values
of the multimodal attributes for such interactor (Feedback = Redundancy, Input =
Equivalence, Prompt = Redundancy). By assigning such properties, which imply the
full use of both graphical and vocal modalities, the navigator interactor includes a
vocal part, with its prompt and feedback, and uses an image link for the graphical
part.
<figure><img src="image_66.png" /><figcaption>Fig. 5. Editing connections among multimodal presentations.</figcaption></figure>

Once the new presentation has been completed we obtain a presentation for the room selection. It contains a grouping with an initial vocal message 'Select the room you want to monitor' to introduce the navigator elements associated with each selectable room. For each navigator there is a vocal prompt that indicates what vocal input to enter to select the corresponding room (e.g. 'Say living to go to the living room'). Figure 6 shows the corresponding user interface implementation.

<figure><img src="image_67.png" /><figcaption>Fig. 6. The multimodal user interface implementation supporting the multiple connections.</figcaption></figure>
## Conclusions and Future Work

This work introduces a novel logical language for multimodal interfaces and the associated environment, which allows designers to easily compose multimodal interfaces and derive X+V implementations. It provides designers with the possibility to work through logical descriptions of the user interface and support for choosing the most suitable combination of various modalities at different granularity levels and for the various parts of the user interface.

This has been integrated in an environment for multi-device interface design and development, thus facilitating the implementation of multiple versions adapted to the various target modalities because of the use of a common abstract vocabulary, which is then refined according to the target platforms. This avoids requiring developers to learn a plethora of details of the many possible implementation languages

This result has been validated through the development of some multimodal applications (one of them is briefly described in the paper), which can be rendered through publicly available browsers (Opera). The authoring environment is publicly available for download of the executable code.

Future work will be dedicated to empirical tests in order to better assess how the development process is facilitated with this approach, especially when multi-device interfaces should be developed (e.g. desktop, mobile, vocal and multimodal versions of the same application).

We also plan to develop an automatic system able to support graphical-to-multimodal user interface content adaptation. Future work will be also dedicated to extending the environment in order to provide support for additional modalities, such as tactile and gestural interaction, in several possible combinations, still for both stationary and mobile devices.

## Acknowledgments

This work has been supported by the EU ICT STREP Project ServFace (http://www.servface.eu/)

## References

1. Axelsson J., Cross C., Lie H.W., McCobb G., Raman T.V., and Wilson L.. XHTML+Voice Profile 1.0. Recommendation, World Wide Web Consortium (W3C), 2001. See http://www.w3.org/TR/xhtml+

2. Calvary, G., Coutaz, J., Bouillon, L., Florins, M., Limbourg, O., Marucci, L., Paternò, F., Santoro, C., Souchon, N., Thevenin, D., Vanderdonckt, J.: The CAMELEON reference framework. CAMELEON project, Deliverable 1.1 (2002).

3. Clark J., Xsl Transformations (XSLT) version 1.0. Technical report, W3C, 1999.

4. Coutaz J., Nigay L., Salber D., Blandford A, May J., Young R., 1995. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: the CARE Properties. Proceedings INTERACT 1995, pp.115-120.
5. Honkala M., Pohja M.: Multimodal interaction with XForms. Proceedings ICWE 2006: 201-208.

6. Lawson J., Al-Akkad A., Vanderdonckt J., Macq B.: An open source workbench for prototyping multimodal interactions based on off-the-shelf heterogeneous components. Proceedings ACM EICS 2009: 245-254

7. Lin, J., Landay, J.A.: Employing Patterns and Layers for Early-Stage Design and Prototyping of Cross-Device User Interfaces. Proc. CHI: 1313-1322 (2008)

8. Myers, B.A., Hudson, S.E., Pausch, R.: Past, Present and Future of User Interface Software tools. ACM Trans. Comput. Hum. Interact. 7, 3–28 (2000)

9. Multimodal Interaction Activity (W3C), http://www.w3.org/2002/mmi/

10. Nichols, J. Myers B. A., Higgins M., Hughes J., Harris T. K., Rosenfeld R., Pignol M., 2002. "Generating remote control interfaces for complex appliances". Proceedings ACM UIST'02, pp.161-170.

11. Obrenovic, Z., Starcevic D., Selic B., A Model-Driven Approach to Content Repurposing, IEEE Multimedia, January March 2004, pp.62-71.

12. Paternò, F., Giammarino F.: Authoring interfaces with combined use of graphics and voice for both stationary and mobile devices. AVI 2006: 329-335

13. Paternò, Santoro, C., Spano, L.D.: MARIA: A Universal Language for Service-Oriented Applications in Ubiquitous Environment. ACM Transactions on Computer-Human Interaction, Vol.16, N.4, November, pp.19:1-19:30 (2009)

14. Paternò F., Sisti C., Deriving Vocal Interfaces from Logical Descriptions in Multi-Device Authoring Environments, Proceedings ICWE 2010, Wien, July 2010, Springer Verlag, LNCS 6189, pp.204-217.

15. Stanciuulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F.: A Transformational Approach for Multimodal Web User Interfaces based on UsiXML. Proc. ICMI: 259-266 (2005)
<figure><img src="image_68.png" /></figure>
# RTME: Extension of Role-Task Modeling for the purpose of Access Control Specification

Birgit Bomsdorf

University of Applied Science Fulda, Marquardstraße 35, 36039 Fulda, Germany
bomsdorf@hs-fulda.de

**Abstract.** Interactive systems are often developed without taking security concerns into account. We investigated a combination of both HCI models and access control specifications to overcome this problem. The motivation of a combined approach is to narrow the gap between different modeling perspectives and to provide a coherent mapping of modeling concepts. The general goal is a systematic introduction and tool support of security concerns in model-based development of interactive system. In this paper we report results of our work currently concentrating on the early design steps. The focus of this presentation is on the specification of task and role hierarchies, conflicting privileges and related tool support.

**Keywords:** Task modeling, Role modeling, Role task assignment, Tool support, Access control.

## 1 Introduction

Task and domain models are commonly used for the purpose of conceptual modeling. The combination of the two describes how users may manipulate objects while performing tasks. Access control management requires similar information, i.e. detailed specifications of the users' privileges to access objects and to perform operations on them. Security, however, is often postponed until the end of the design cycle or until the implementation of a system [4], [12]. Interactive systems are therefore developed without taking authorization concerns into account.

An access control model defines the permissions of users (e.g. human users, processes, computers) to access system resources (e.g. on an object, data base content, a file). Role is the central concept of prevalent access control models (Role-Based Access Control, RBAC [10]). Approaches such as [6], [8], [15] extending RBAC as well as the task-modeling approach TADEUS [13] have been showing the demand of differentiating between roles based on the structure of an organization (organizational roles) and roles based on the privilege to perform tasks (task-grouping roles).

In HCI this distinction exists but is hardly introduced into the kernel concepts of tools supporting task-based modeling. In some tools an explicit role model does not exist (K-MADE [1], Diane+/Tamot [7], TaskArchitect [14]). In contrast to this, in CTTE [9] separate task models are created per role defining all tasks that can be performed by that role. The role specification, however, does not support inheritance of privileges. This is supported in WSDM [3] but resulting role hierarchies are not
formally integrated with task models. All in all, concepts such as agent, actor, role, and group are used equivocally and ambivalently even within a single HCI approach. Furthermore, (semi-) automatic support in modeling the group, role, and task hierarchies taking care of their mutual dependencies is hardly supported. These facts complicate the combination of HCI modeling with security concerns.

Guo [5] has been showing in his work how the complexity of the three hierarchies and their relationships may be handled in the context of access rights. Based on his proposal we extended our modeling approach [2]. The extensions are also combined with MAD [1] aiming at modeling extensions in general. Therefore, for the remaining of the paper we refer to it by the abbreviation RTME (Role-Task Model Extension). In the following, as indicated by the name, the focus is on tasks, roles and their mutual dependencies. The approaches reported in [12] and in [4] are comparable with our work. They aim at the integration of access control specifications with models known from Software Engineering and Web-Engineering, respectively. However, privileges are formulated by means of roles and system functionality, not considering the context of users performing tasks to reach goals.

# 2 RTME: Integrative Modeling

Fig. 1 depicts an overview of the model and the interrelations of our integrative approach. It enables the explicit specification of organizational roles, named *groups*, as well as of task-oriented *roles* to define privileges. Users (individual persons) are specified as well. They are members of groups whereby, because of *adopts* relations, they are enabled to perform tasks (*performs* relation) and hence to act on objects (*involves* relation), i.e. to invoke methods. Our current tool, which is based on these concepts, supports the creation of group, role, and task hierarchies taking into account their interdependencies. Underlying functions check consistency violations and support to solve them. Hereby they contribute to the reduction of modeling complexity and to avoid modeling errors.

<figure><img src="image_69.png" /><figcaption>Fig. 1. Overview of modeling concept</figcaption></figure>

## 2.1 Task Model

Figure 2 depicts a task model example by means of the notation as introduced in MAD [1]. The simplified task model consists of three unconnected task trees. The task *online shopping* is a composition of the subtasks *rate seller* and *buy product*, which is further refined. The task *sell online* is subdivided into two subtasks, whereas *administrate website* simply consists of a single task. Additional concepts are commonly in use to specify the order of task execution, such as temporal relations, and conditions constraining task performance, e.g. pre- and post-conditions. These
concepts are not detailed here since the subsequent considerations reference only the
task hierarchy. The following remarks are added only to complete the description of
the example: The sequencing of the direct subtasks of *online shopping* as well as of
*sell online* is open because of the *No order* declaration in each case. The option
*enabling* defines that subtasks are to be performed one after the other, in which the
sequence is given by the graphical order from left to right in the diagram. *Elementary*
is used for leaf tasks, i.e. for tasks without subtasks. The label *OPT* denotes optional
task execution.

<figure><img src="image_70.png" /><figcaption>Fig. 2. Example of a task and a role model</figcaption></figure>

## 2.2 Role Model

A role-task-mapping “r performs t” (see figure 1) means that the user who has taken
role r possesses the privilege to perform t. The set of all privileges of a role r is
denoted by *privileges(r)*. Roles are structured by a so called poly-hierarchy that is
given by a graph. While the task hierarchy expresses composition relations, the role
hierarchy describes inheritance of privileges. An edge from a role r1 to a role r2
indicates that the privileges assigned to role r1 are a proper subset of the privileges of
r2, i.e. *privileges(r1) ⊂ privileges(r2)*. For this kind of relation it is said that r1 is
junior to r2 and that r2 is senior to r1 [10].

In our example we want to allow each person to browse the catalogue. However,
only persons adopting a role *buyer* should be enabled to buy a product and to rate a
seller. First of all we create a role *everyone* and assign to it the task *browse catalogue*.
Fig. 3 left hand shows the result of this editing step. Inserting the first role is done
easily. It is positioned between *minRole* and *maxRole* as shown in the example with
*privileges(minRole) ⊂ privileges(everyone) ⊂ privileges(maxRole)*.

Each graph posses a minimal and a maximal role that are introduced for the
purpose of computing a role model’s hierarchy and taking care of conflicts. A detailed
description is provided in [5]. Please note that the set of privileges of *minRole* is
always empty while *maxRole* possesses all the time all privileges defined by the task model.

Now we create a role *buyer* and assign the task *online shopping* and hereby also all of its subtasks to the role (see *Inherited role task(s)* in the role editor window). This step results in a model (see Fig. 3 bottom), in which *buyer* is positioned into the role hierarchy according to the privileges added to the role as well as to the hierarchy existing so far. The task *browse catalogue* is part of *online shopping*, i.e. *privileges(everyone) ⊂ privileges(buyer)* holds true, based on which the position is determined. Similarly the roles *seller* and *administrator* are inserted to define privileges for *administer web site* and *sell online*. These two tasks are not connected to the tasks taken into account so far. Hence, both roles are inserted separately into the role hierarchy (see figure 5 right hand). Explanations of more complex cases can be found in [5] and [11].

<figure><img src="image_71.png" /><figcaption>Fig. 3. Inserting the role *buyer*</figcaption></figure>

## 2.3 Conflicts

Task models as well as role models mostly lead to complex hierarchical structures. Complexity is increased by their mutual dependencies. Thus, different conflicts within a specification may result from assigning roles and tasks to each other. The existing hierarchical structures of both the tasks and the roles are to be considered. It is not allowed to assign more tasks to a junior role than to one of its senior roles. This would result in an *assignment conflict* because in such a case *privileges(senior role) ⊂ privileges(junior role)* would hold true. This, however, is conflicting with the definition of the role hierarchy (given above). A strategy for avoiding such modeling errors is incorporated in RTME. Each time a user of our role editor selects a task that would cause a conflict a warning is shown and the user is prompted to perform a
correction. Similarly RTME evaluates editing steps while the task model is under construction or is being modified.

In addition to such *assignment conflicts* RTME enables to define explicit cases of conflicts, named *privilege conflicts* and *role conflicts*, which are checked during modeling.

### 2.3.1 Role Conflicts

A *role conflict* is defined for two roles r1 and r2, formally denoted by $r1 \leftarrow r2$. It specifies that roles are mutually exclusive. In such a case a user must not take both roles; it is also forbidden to assign the two roles to the same group. Hence, either r1 can be assigned to a group g or r2 can be assigned to g. Furthermore, for $r1 \leftarrow r2$ it must hold true that no path exists between r1 and r2 in the role graph.

Basically, the two roles r1 and r2 must be independent from each other, i.e. the only common junior is *minRole* [5]. (A role is a common junior role of two roles if it is junior of both.) In addition, *maxRole* must be the only senior role of the two conflicting roles r1 and r2 [5]. Thus, a role conflict can only exist if r1 is neither senior nor junior to r2. Otherwise, r1 and r2 would have at least a common privilege violating the mutual exclusiveness.

Obviously, a defined role conflict does not only impact group but also task modeling. Fig. 4 shows two role conflicts specified for the example given above: *seller* $ightarrow$ *administrator* and *buyer* $ightarrow$ *administrator*. Hence, common privileges of the roles *seller* and *administrator* as well as of the roles *buyer* and *administrator* are mutually exclusive. The attempt, for example, to define the task *administrate web site* as a subtask of *online shopping* results in an error now. Please note that the task *online shopping* is assigned to the role *buyer* while *administrate web site* is assigned to *administrator*. In the case *administrate web site* should become a subtask of *online shopping*, the role model would have to be modified as well so that a junior relation exists between *buyer* and *administrator*.

<figure><img src="image_72.png" /><figcaption>Fig. 4. Role conflict example</figcaption></figure>
### 2.3.2 Privilege Conflicts

In addition to formulating conflicts between roles RTME enables to define conflicts of privileges. A privilege conflict specifies two tasks t1 and t2, noted by $t1 < > t2$, that must not be assigned to a role r at the same time. The role maxRole is an exception to this rule as it comprises all privileges existing in the model. However, this exception causes no problem since maxRole cannot be assigned to any user or group of users.

<figure><img src="image_73.png" /><figcaption>Fig. 5. Specification of a privilege conflict.</figcaption></figure>

Fig. 5 shows a privilege conflict defined for the tasks *rate seller* and *sell online* (*rate seller* < > *sell online*). Hence, *rate seller* and *sell online* must not be assigned to the same role. The role editor in Fig. 6 shows the attempt to allow a user who takes the role *seller* to rate sellers. However, the task *sell online* has been assigned to *seller* and thus the set *privileges(seller)* cannot be expanded by *rate seller*. RTME forbids the modification showing an error message. It is up to the developers to decide on corrections. For example, they might remove the privilege conflict (*rate seller* < > *sell online*). In the same way *privileges(buyer)*, that already contains *rate seller*, cannot be extended with *sell online* (see for privileges of buyers the role editor content shown in Fig. 3).

<figure><img src="image_74.png" /><figcaption>Fig. 6. Privilege conflict impacting role modeling.</figcaption></figure>

The defined privilege conflict has a similar impact on task modeling. The task *sell online* cannot be specified as a subtasks of *rate seller* (see Fig. 7). If the *rate seller- sell online* conflict is not specified the role seller would become a junior role of *buyer* because of the automatic role structure computation.
<figure><img src="image_75.png" /><figcaption>Fig. 7. Privilege conflict impacting task modeling.</figcaption></figure>

3 CONCLUSION

The modeling steps presented in this paper basically consist of privilege definitions. Hereby the groups a person belongs to, the roles a person may adopt, the tasks a person can perform, and the objects a person is allowed to access are specified. The definition of group structures, similarly to role modeling as presented above, is done by subset relations (also resulting in an acyclic graph with directed edge). RTME handles the specification of group hierarchies and dependencies on role models in a very similar way as role-task modeling.

The underlying theory combines our own work on task modeling [2] with the work on conflict handling by [5]. Hence, our approach provides not only a sound theoretical basis but contributes also to integration of HCI with Access Control. The extensions with respect to our previous work are twofold: On the one hand the differentiation of organizational groups and roles defining privileges is incorporated. On the other hand RTME implements an integrative group, role and task modeling approach. RTME assists in structuring the model taking into account model properties defined so far. Each time a modeler inserts an additional assignment the resulting hierarchies are determined and the new assignment is inserted only if it causes no conflicts. This technique allows, for example, assigning roles and tasks to each other without performing required restructuring of the hierarchies. RTME evaluates instead the new graph models considering resulting and explicitly defined rules of conflicts. This approach meets the fact that the model hierarchies result from the groupings of privileges.

Construction of model hierarchies and conflict control is based on the rules defined for the meta-model (assignment conflicts) together with the conflict rules defined by the modeler, namely role conflicts and privilege conflicts.

In the examples modeled so far by means of RTME the underlying algorithms have proven to be very useful. However, the models were relatively small. We are aware that in the context of real projects more support is needed to reduce complexity and mental load. A common technique is to provide various views on a model in conjunction with filtering mechanisms. The first implementation of RTME [11] provides such view generation. The modeler can, for example, extract all objects a
special person has access to or all persons that are allowed to take a specific role. The interactive extracted views are presented by a diagram similarly to those used during editing.

The RTME editor is in the state of a prototype. It possesses import and export functions (on the basis of XML) enabling the exchange of models with other tools. Currently our own task modeling approach and MAD are supported. Generally, RTME can be combined with task models in which a super task is the sum of its subtasks, i.e. the superior task does not define additional functionality.

**Acknowledgments.** The author likes to thank Andreas Reitschuster for his contribution to this work.

## 4 References

1. Baron, M., Scapin, D.: K-MADE User Manual, http://kmade.sourceforge.net, (accessed 22.8.2010) (2006)

2. Bomsdorf, B.: The WebTaskModel Approach to Web Process Modelling. In: TAMODIA Workshop, Springer, LNCS 4849, 240-253 (2007)

3. Casteleyn, S.; De Troyer, O.: Structuring Web Sites Using Audience Class Hierarchies, In: DASWIS 2001 workshop (in conjunction with the ER 2001 conference), 198-211 (2001)

4. Díaz, P., Aedo, I., Sanz, D., Malizia, A.: A model-driven approach for the visual specification of Role-Based Access Control policies in web systems, In: Visual Languages and Human-Centric Computing, VL/HCC 2008, 203 – 210 (2008)

5. Guo, Y.: User/Group Administration for RBAC. The University of Western Ontario (1999)

6. Kang, M.H.; Parker, J.; Forscher, J.N.: Access Control Mechanisms for Inter-Organizational Workflow, In: Symposium on Access Control Models and Technologies, 66-74 (2001)

7. Lu, S., Paris, C., Vander Linden, K.: Tamot: Towards a Flexible Task Modeling Tool. In: Proceedings of Human Factors, 878-886 (2002)

8. Osborn, S., Nyanchama, M.: The Role Graph Model and Conflict of Interest. The University of Western Ontario: s.n., (1999)

9. Paternò, F., Santoro, C. & Tahmassebi, S.: Formal Models for Cooperative Tasks: Concepts and an Application for En-Route Air Traffic Control. In: Design, Specification, and Verification of Interactive Systems, Springer-Verlag, Abingdon, 71-86 (1998)

10. RBAC Standard, http://csrc.nist.gov/rbac/rbac-std-ncits.pdf, (accessed 22.8.2010) (2003)

11. Reitschuster, A.: Realization of an Editor supporting an integrative Modeling of Groups, Roles and Tasks (Realisierung eines Editors zur integrativen Gruppen-, Rollen- und Aufgabenmodellierung, in German), Masterthesis, University Hagen (2008)

12. Romuald, T., Stéphane, C.: Integration of Access Control in Information Systems: From Role Engineering to Implementation, In: Informatica 30, 87-95 (2004)

13. Stary, C.: Role-Adapted Access to Medical Data: Experiences with Model-Based Development, In: Universal Access in Health Telematics, 224-239 (2005)

14. Stuart, J. & Penn, R.: TaskArchitect: taking the work out of task analysis. In: 3rd Annual Conference on Task Models and Diagrams, TAMODIA 2004, 145-154 (2004)

15. Zhang, C., Hu, Y., Zahng, G: Task-Role Based Dual System Access Control Model, In: International Journal of Computer Science and Network Security, 6(7B), 211-215 (2006)
# Web Applications Usability Testing With Task Model Skeletons

Ivo Maly, Zdenek Mikovec,

Czech Technical University in Prague, Faculty of Electrical Engineering,
Karlovo namesti 13, 121 35 Prague, Czech Republic
{malyi1, xmikovec}@fel.cvut.cz

**Abstract.** Usability testing is technique for analysis of the usability problems of applications, but it requires significant effort to prepare the test and especially to analyze data collected during the test. New techniques such as usage of task models were introduced to improve and speed up the test analysis. Unfortunately, only few applications provide the task model. Therefore, we propose a method and tools for partial reconstruction of the task list and the task model called skeleton. This reconstruction is done from the usability expert's application walkthroughs. The task model skeleton is generated automatically, but it should provide similar information during the usability data analysis as manually created full-scaled task model. In order to evaluate usage of the task model skeleton we conducted a usability study with the web e-mail client Roundcube. Results show that the task model skeleton can be used as a good substitution for the manually created task model in usability testing when full-scaled task model is not available.

**Keywords:** Usability testing, Task list, Task model, Web applications

## 1 Introduction and Motivation

Incorporating usability testing into software application development process, as presented e.g. in [2], can significantly increase the efficiency of the development process and the acceptance of the final application by the users. The problem is that usability testing is not an easy and straightforward process. It consists of several steps, see steps rectangles in Fig 1, which are time consuming. In each step of the usability test wide range of supportive data and documents are created, e.g. task list, screener, various questionnaires and forms for annotations, test logs. Also, additional data can be recorded like audio/video recordings or status log of the tested application. This information is not always interconnected, e.g. task list and audio/video recordings collected during the execution of the usability test. These not properly interconnected data is hard to analyze in the Step 4 of the usability testing process. If the data is interconnected sufficiently (e.g., usage of task model to create relations between collected data seem to be a very promising approach), we can use more sophisticated analytical methods, e.g. statistical [6] or visual analysis [1], to find usability problems of the tested application.
<figure><img src="image_76.png" /><figcaption>**Fig. 1.** Schematic diagram of the usability testing process with the task model. In the lower part there are steps of the usability testing process and in the upper part there are the task list and the task model incorporated into the process of usability testing.</figcaption></figure>

## 1.1 Usability Testing with Task Models

In order to interconnect the collected data the task model can be used. In Fig. 1 we can see that the task model is created prior the start of the usability test and it is used in Step 3 during the execution of the usability test and mainly in Step 4 during the analysis of the data collected from the usability test. In Step 3, we also use task list, which is the foundation of the usability test. Usually, the task list and the task model are created separately, even though there is relation between them. During the execution of the usability test the task model can be used for interconnection of observer's annotations with the task, as presented in [1]. It can be also used for post-test interconnection of log records generated during the remote usability test with the task from the task model, as presented in [6].

Main benefits of the task model usage can be found in the analysis (Step 4), where it is used as an interconnection between the task list and the collected data. For each user interaction collected we can judge whether it belongs to the currently solved task or not without any time consuming search in the video recording or in other data sources. Several usability test analysis tools, which take advantage of task models, were introduced in [1], [4] and [5]. They show collected data in the form of timelines visualizations in order to present the length of each task, length of the interaction and they give information whether the interaction was expected in the current task.
## 1.2 Issues of Usability Testing with Task Models

Problem of usage of task models in the usability tests is the necessity to have the task model at disposal during the test. There are 2 typical scenarios that may happen:

* **Task model is available.** This is typical for applications that were generated automatically or semi-automatically based on the task model. Therefore the task model was created during the design of the application.

* **Task model is not available.** If we want to use the task models, we need to create them. Creation of the task model may not be an easy process, due to the often complexity of the task or the tested application. Creation of the task model is also additional activity that must be performed before the test begins.

For the majority of applications the task model is not available at the time of the usability testing. Therefore, we were looking for the possibility of automatic or semi-automatic generation of the task model or similar data structure, which would have similar properties like the task model during the usability test execution and during the analysis of the data from the test (Step 3 and Step 4 in Fig. 1). Our solution, depicted by dashed lines in Fig 1, is based on the expert walkthroughs that are created during the task list creation step (Step 2.2 in Fig. 1). From these walkthroughs the *task list skeleton* and the *task model skeleton* are created instead of the task list and the task model. The task list skeleton is a template for the task list with the list of expected steps. Details of the task list skeleton are in chapter 2.3. The task model skeleton is a sequence of expected user interactions. Compared with the task model, the task model skeleton is much simpler, without task relations or without hierarchical structures. Details of the task model skeleton are in chapter 2.4.

# 2 Creation of Task List Skeleton and Task Model Skeleton for Web Applications

Presented approach of creation of the task list and task model skeletons is on one hand generic but on the other hand it may differ on each application platform (e.g. web, desktop). In this paper, we will focus on the tools for the web application testing and on creation of the task list and the task model skeletons for web applications.

## 2.1 Related work

Similar approach of automatic task model creation was presented by Paganelli et al. [8]. Authors parsed the HTML structure of each web page and created a Concur Task Tree (CTT) task model for the whole web application. Problem of the presented approach is that it generates quite big task models even for quite simple web applications, e.g. application with about 10 web pages is represented by CTT task model with 181 states. Such huge tree is good for computer processing but will be complicated for human expert. Another drawback is that it follows the HTML
interactive objects only. Current rich web applications are using JavaScript to provide
interaction with the user. Therefore the presented algorithm will not detect parts of the
interactions and task model states based on the JavaScript. While our approach is
based on recording of interactions executed by an expert walkthrough we are able to
record all of them. Also the task model is containing much smaller number of states
that make it easily understandable.

## 2.2 Interaction Log Recorder

Our approach of automatic task list and task model reconstruction is dependent on logging of user interaction (log record). Logging is used to detect and save both expert walkthroughs in Step 2.2 and user interactions during usability test (Step 3). Each log record must provide sufficient information about the type of interaction performed and about the new state into which the application had moved. In this work we were focused on implementation of a log recorder for web applications. Our recorder is based on Selenium IDE (http://seleniumhq.org/projects/ide/), which is a browser plug-in using JavaScript to listen to mouse clicks and key presses and records the data about interaction. Recorder is application independent so we do not have to install custom JavaScript code into the tested application in order to record interaction log. The structure of the log record is standard Selenium IDE XML log format [9]. While the standard Selenium IDE plug-in does not store time stamps, we have implemented custom function to include them into the log record.

## 2.3 Task List Skeleton

The task list skeleton is a template of a task list with the list of expected steps. It is a rich text document where expected steps generated by the expert walkthrough are automatically inserted into task list template using XML transformations. If there are more walkthroughs for the particular task, these walkthroughs are transformed into separated lists of the expected steps. In the Table 1, there is an example of the task definition from the usability test of the Roundcube e-mail client. Details about the usability test are in the chapter 3. The task description (in italics) must be entered by the usability expert who prepares the usability test. When the missing parts of the document are filled in the document becomes finalized task list and can be used in usual way by the usability test moderator and logger.

## 2.4 Task Model Skeleton

The task model skeleton is a sequence of expected user interactions. Visual representation of the task model skeleton automatically generated for the Task 5 (expert walkthrough 5.1) is in Fig. 2. There are 3 interactions that must be executed in order to complete the task. In Fig 3, the manually created task model for the Task 5 is presented, which is based on two expert walkthroughs (walkthrough 5.1 and walkthrough 5.2). Compared with the full-scaled task model, the task model skeleton
is much simpler, without task relations or without hierarchical structures. Each subtask of the task model skeleton reflects one step in the task list.

<table><caption>**Table 1.** Example of the task list skeleton for the task 5 from the Roundcube e-mail client usability test.</caption><thead><tr><th></th><th>Task list skeleton</th></tr></thead><tbody><tr><td>Task description</td><td>Open first new message in inbox and read it. Add sender of the message to the address book.</td></tr><tr><td>Expected steps</td><td>1. Click at the link "Info bulletin".<br>2. Double click at the link "Info bulletin".<br>3. Click at the image "add".<br>-- or --<br>1. Click at the link "Info bulletin".<br>2. Double click at the link "Info bulletin".<br>3. Click at the "rcbtn101". (Address book)<br>4. Click at the "rcbtn105". (New contact)<br>5. Type "Info" into field "rcmfd_name".<br>6. Type "info@lkom.cz" into field "rcmfd_email".<br>7. Click at the "rcbtn100".</td></tr></tbody></table>

<figure><img src="image_77.png" /><figcaption>**Fig. 2.** Visual representation of task model skeleton 5.1 for task 5.</figcaption></figure>

Each subtask of the task model skeleton contains the same information about the user interaction as was recorded by the interaction log recorder. Thanks to that we can perform comparison of the subtask in task model skeleton with particular interaction in user log record in order to match the interaction with the subtask. Task model skeletons are stored in XML file format in order to be easily loaded by the analytical applications used in analysis step (Step 4 in Fig 1).

<figure><img src="image_78.png" /><figcaption>**Fig. 3.** Task model for task 5 representing possible task model created from the task model skeletons 5.1 (in Fig 2) and skeleton 5.2</figcaption></figure>
# 4 Use Case: Web E-mail Client Roundcube

In order to evaluate our proposal of the web application usability testing with the task model skeletons, we executed the usability test of the web e-mail client Roundcube (http://roundcube.net). At the beginning, we have analyzed the application and created the expert walkthroughs using the interaction log recorder (see chapter 2.2). Task list and the task model skeleton were created using XML transformations and the task descriptions in the task list were added. During the test we have recorded the user interaction using the interaction log recorder and then we have analyzed the data using Interactive Visualization Environment (IVE) tool [7]. IVE is an interactive tool for visual analysis of data from usability studies. IVE uses an internal object database and convertor plug-ins to convert collected usability test data into the internal database. Usability expert uses set of interactive visualization views to analyze the data. Each interactive visualization view is developed as a plug-in that has access to the IVE internal database and it can communicate with other plug-ins through a simple message dispatching system

## 3.1 Test Setup, Task Model and Task List Skeleton Creation

According to a minimal required number of participants for usability study [3] we selected 6 participants, both technically experienced and non-technical ones. None of them had a previous experience with the Roundcube client, but some of them had experience with other web e-mail clients, such as Gmail. The test was conducted in the usability lab and including pre- and post-test questionnaires it took from 30 to 40 minutes. Beside the data collected by the interaction log recorder we have also recorded an audio and video from each session.

During the application analysis we have selected 11 tasks and prepared the task model skeletons and the task list skeleton for them. Tasks were focused on typical e-mail activities like reading, sending and forwarding e-mails. We have also focused on e-mail folder management and work with address book. Each task in task list and task model skeleton was created using one expert walkthrough except the task 5, where we have recorded two walkthroughs. Creation of the task model skeleton and the task list skeleton generated only a small time overhead over the standalone task list generation, because we needed to save recorded interactions for every task, which has slightly broken the flow of recording process. Also recording of the alternative walkthrough for task 5 required rollback of the system to the previous state that took few minutes. Overall process of the task model skeleton generation and the task list skeleton generation was perceived positively.

## 3.2 Analysis of Usability Test Results with help of Task Model Skeleton

In the Fig. 4, there are two timelines representing data collected during the usability study of the Roundcube e-mail client of the participant p7. The lower timeline shows length of each task. The upper timeline shows recorded log interaction combined with task model skeleton. Each rectangle represents one user interaction. The color of the
rectangle highlight, whether the interaction was correct (green), incorrect (white) or it
may be correct, because the interaction is expected in some moment in the task
(yellow). The interpretation of the sequence of various colors is:

* **Sequence of green rectangles.** Such a sequence represents optimal execution of the task (e.g. Task 6). There may be white rectangles before the first green rectangle (e.g. Task 2), which represent unexpected interaction before the optimal execution started.

* **Sequence of green rectangles, which is interrupted by white rectangles and continues with green rectangles.** Such a sequence represents an event, but the participant performs optimal execution of the task, gets lost, but he/she is able to recover back.

* **Sequence of green rectangles, which turns into sequence of yellow rectangles.** Such a sequence represents the same start as in the previous example, but the participant recovers with interaction from the task but not the expected one (e.g. user starts from beginning).

* **Sequence of yellow rectangles.** Such a sequence (e.g. Task 3) represents a situation when the participant starts the execution of the task with interaction that is part of the particular task model skeleton but is not the expected one. Example of such a case is usage of browser navigation buttons or optional execution of the first (or expected) interaction.

The interpretation of the timeline visualization allowed us to focus on the most
problematic tasks, e.g. in task 3 we have found that the participant performed search
action from different web page and in the task 5 the participant added wrong contact.
In the task 2, the participant was not sure if he/she finished the task, so he/she
performed last interaction again.

<figure><img src="image_79.png" /><figcaption>Fig. 4. Visualization of the timeline for participants 7.</figcaption></figure>

4 Conclusion

In this paper we have discussed the advantages of the usability testing with the task
models. We pointed out that the biggest barrier of this method is the missing task
model for the most of the tested applications. To overcome this barrier we have
suggested the modification of the task list creation step be extended to create also the
task model. The result is the automatically created task model skeleton and the task
list skeleton. These are then used for task model based usability testing and analysis.

We have conducted the usability study of the web e-mail client Roundcube with 6
participants in order to evaluate our concept of the task model skeleton. The analysis
of the data showed that we were able to match user's interactions with task model
skeleton and to show which interaction belonged to which task. Then we showed the