# Approaches to Software Engineering: A Human-Centred Perspective

Liam J. Bannon

<figure><img src="image_1.png" /></figure>

## To cite this version:

Liam J. Bannon. Approaches to Software Engineering: A Human-Centred Perspective. Third IFIP WG 13.2 International Conference on Human-Centred Software Engineering (HCSE), Oct 2010, Reykjavik, Iceland. pp.1-5, 10.1007/978-3-642-16488-0_1. hal-01055200

HAL Id: hal-01055200

https://inria.hal.science/hal-01055200

Submitted on 11 Aug 2014

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Approaches to Software Engineering: A human-centred perspective

Liam J. Bannon

Interaction Design Centre
Dept. of Computer Science & Information Systems
University of Limerick
Limerick, Ireland
Liam.bannon@ul.ie

**Abstract.** The field of software engineering has been evolving since its inception in 1968. Arguments as to the exact nature of the field, whether it should be conceived as a real engineering profession, the role of formal methods, whether it is as much an art as a science, etc., continue to divide both practitioners and academics. My purpose here is not to debate these particular topics, but rather to approach the field from the outside, coming as I do from a long period of involvement in the human and social side of the computing discipline, namely, from the fields of Human-Computer Interaction, Computer Supported Cooperative Work, Participative Design, Interaction Design, and Social Informatics, more generally. I wish to examine how this “human-centred” perspective might shed a new light on some issues within the SE field, perhaps opening up topics for further discussion and examination.

**Keywords:** CSCW, human-centred computing, requirements, sociology, software engineering

## Extended Abstract of the Keynote

It is difficult to talk about issues in the Software Engineering (SE) field without first noting the larger landscape of computing and information systems in which it is embedded. Computing traditionally has focused on answering the question: What can be automated? (e.g. Arden, 1980). While the term computer was originally used to describe real people performing numerical calculations, the human side of computing has tended to be ignored within the emerging discipline of computer science, which has focused on hardware and software issues. Emphasizing this, one of the first professional organizations for people involved in computing was titled The Association for Computing Machinery (ACM). As focus has shifted from mainframe computing to personal and now ubiquitous computing, there has been a slowly increasing awareness of the need to pay greater attention to the human aspects of computing. This implies much more than simply noting the social implications of computing technology, but rather seeks to view the activities of people involved in
various aspects of computing, especially systems development and programming, as a
legitimate, and necessary part of a computing curriculum.

Many people have been involved in the attempt to shift the focus of computing -
and informatics more generally – away from a purely technical approach concerned
with hardware and software only, to one that considers the human activities of design
and use of information systems as being of central concern. Interestingly, many of
these people have come from the Nordic countries. My own selection of pioneers in
this space would include people such as Kristen Nygaard, who argued for a
perspective on systems development that included the social and political, as well as
the technical. People like Peter Naur, whose compilation of papers was published by
ACM under the title Computing: A Human Activity, which emphasized the human
side of programming and systems development. People like Christiane Floyd, from
Germany, who presciently wrote of different paradigms in software engineering and
the need to allow for multiple perspectives in the field. In the US, perhaps one of the
earliest popular publications that promoted a human-centred approach to software was
the 1971 book by Gerry Weinberg, a practitioner and consultant, entitled The
Psychology of Computer Programming. Rob Kling spent many years as an advocate
of a more open computer science discipline that he labelled “Social Informatics”. In
recent years, a number of senior figures in the field have also put their hats in the ring:
Peter Denning, former President of ACM, arguing for a new and more expansive
computing profession; Denis Tsichritzis, head of GMD, the former German national
research centre for IT, critiquing much old-fashioned computer science as being akin
to “electric motor” science; Peter Wegner, in theoretical computer science, arguing
that the concept of interaction in computing is fundamentally more powerful than
algorithms; and Terry Winograd, one of a number of people involved in bringing the
larger field of Design into computing, and developing the Interaction Design field.
All of these authors, despite significant differences in their messages, to my mind
share a critique of how the field of computing and the academic discipline of
computer science has been defined, circumscribed, and taught to students, and all
advocate a more “human-centred” approach, in one form or another. For example, in
reflecting on our educational system, Denning (1992) notes: “A curriculum capable of
preparing students for the shifting world must incorporate new elements emphasizing
design, demonstrated proficiency, effective interaction with others, and a greater
sensitivity toward the historical and cultural spaces in which we all live and work”.
The issue here is not simply providing computer science students with a rounded
education, but more fundamentally questions the very nature of the discipline, arguing
that human activities and interests are part of the core of the computing discipline,
whenever we conceptualize, design, build, and test new technologies. It is this
tradition that I wish to discuss in the context of human-centred software engineering.

These alternative views of the computing field have, I believe, contributed to the
slow emergence of what is beginning to be termed, in some quarters, “human-
centred” computing (HCC). The label may appear somewhat meaningless, as who
would subscribe to an alternative “system-centred” computing label? However, just
as the label “user-centred design” in the field of human-computer interaction hit a
chord in the 1980’s, it may be the case that the “human-centred computing” label will
have similar re-orienting effect on the field of computing today. Likewise with other
new terms that are appearing currently. For example, the emergence of new terms
and research areas, such as the “new informatics” to augment traditional information
systems research, and “interaction design” augmenting traditional HCI, are, in my
opinion, examples of shifts in perspective towards a more wholistic view of human-
systems interaction that begins to pay more attention to the inextricable inter-weaving
of the human, social and cultural with the technical aspects of computing. Note that
these are not simply surface changes, nor should they be viewed simply as ancillary
issues in relation to the dominant computational approach, but rather they raise
foundational issues for the field of computing per se. While this is not the place to
further develop this argument, I wish now to briefly examine how this human-centred
perspective, loosely described above, might be of interest within the software
engineering field. The primary area I will focus on my keynote is in the requirements
engineering phase of software development.

Early textbooks on software engineering provided scant coverage of any “human” issues, with perhaps a brief mention concerning meetings with user representatives in the derivation of requirements, and in designing the user interface. However, we can observe an increasing concern with “user issues” in standard SE textbooks over the years. The increasing prominence of Participative Design approaches to system development, involving close cooperation with users in all phases of an iterative design process, and the prominent role of prototyping and testing, was starting to be felt in the HCI arena in the late 80’s. Also, the rise of the CSCW field was occurring at this time. The CSCW area brought in researchers from other human sciences than psychology, such as sociology and anthropology, to better understand the everyday lives of people, with a view to providing insights that might be useful in the design of more habitable systems. In the case of the classic Sommerville (2010) text on SE, this can clearly be linked to the rise of the CSCW field and the establishment of a CSCW Centre at Lancaster where sociologists and software engineers were involved in joint projects. However, as I will detail in the keynote, this marriage of social and computing science has not been without some difficulties, especially in the context of “producing requirements”. There is an issue as to whether the developing relations between such unlikely bedfellows as technical systems developers and social scientists, particularly ethnographers, and more narrowly ethnomethodological ethnographers, should be seen as a virtuous coupling or a “deadly embrace”. While it should be obvious that I am in favor of any and all approaches to requirements that open-up this phase to a richer appreciation of the work context and work practices of people, I also feel that this recent courtship between developers and sociologists may turn sour due to a misalignment of motives and interests. If we are to have a useful interplay between these two professions then perhaps we also need to be aware of their different agendas, so as to reduce confusions and misunderstandings. I will explore this issue in greater detail in the keynote.

Returning to this issue of “requirements” in SE, one finds a number of perspectives
on them, as evidenced by the different language used. So, for some people, systems
design begins with the need for “requirements capture” - which to me inspires an
image of requirements as well-defined entities just waiting to be plucked from the
environment. It goes without saying that this particular viewpoint is less widely held
today than heretofore. A less extreme view, yet one which is still quite popular in the
engineering community is the notion of requirements "gathering", which again has an
implicit, if not explicit, conception of requirements as things that are waiting to be
harvested. Continuing on this line, one can hear discussion of requirements
"elicitation" which begins to acknowledge that requirements may not be immediately
apparent, or accessible, and may require some effort to "bring forth" from the user
community. Going one step further, we can argue that requirements are not "out
there" awaiting collection, but are themselves constructions, jointly and severally
produced by a range of actors, including users and developers in specific contexts of
discussion, observation and analysis. This view thus requires that we pay close
attention to the ways in which we investigate the use situation and work context, and
take into account the social, political and economic factors involved in the
requirements process. (In this regard, the edited collection by Jirotka and Goguen
(1994) provides an interesting range of positions on social and technical issues in
requirements engineering.)

A number of commentators have noted how requirements as fixed “texts” can
impede a good design process. The designer Chris Jones (1988) argues: “...[we must]
recognize that the ‘right’ requirements are in principle unknowable by users,
customers, or designers at the start.” This position calls into question the nature of
most formal software development contracts today. Similarly, the consultant Tom
Gilb (1990) stresses the need to focus on process, not method or static product. He
notes that current development methodologies “...are based on a static product model.
They do not adequately consider our work to be a continuous process—derived from
the past and being maintained into the future.” Yet another voice in support of this
shift, coming from academic software engineering, is that of Floyd (1987). She argues
for more emphasis on the process of software development than on the efficiency of
the resulting code: “The product-oriented perspective regards software as a product
standing on its own, consisting of a set of programs and related defining texts...
considers the usage context of the product to be fixed and well understood, thus
allowing software requirements to be determined in advance,” while the process-
oriented perspective “views software in connection with human learning, work and
communication, taking place in an evolving world with changing needs... the actual
product is perceived as emerging from the totality of interleaved processes of
analysis, design, implementation, evaluation and feedback, carried out by different
groups of people involved in system development in various roles.” It is interesting
that some of the recent moves to Agile Methods in software development and the rise
of the Extreme Programming movement would seem to provide support to aspects of
the above viewpoints, and thus show, in some respects, a focus on a more “human-
centred” approach.
References

1. Arden, Bruce W. (Ed.) (1980) What can be Automated? The Computer Science & Engineering Research Study (COSERS). MIT Press series in Computer Science: 3, Cambridge, Mass.: MIT Press.

2. Denning, Peter (1992) Educating a new engineer. Communications of the ACM, vol. 35, no.12, December, pp.83-97.

3. Floyd, C. (1987). Outline of a paradigm change in software engineering. In G. Bjerknes, P. Ehn, & M. Kyng (Eds.), Computers and democracy – A Scandinavian challenge (pp. 191-212). Aldershot, UK: Avebury.

4. Gilb, T. (1990). Project Management for the 1990s. The American Programmer, 16-30.

5. Jirotka, M. & J. Goguen (Eds.) (1994) Requirements Engineering: Social & Technical Issues. London: Academic Press.

6. Jones, J.C. (1988). Softecnica. In J. Thackara (Ed.), Design after modernism: Beyond the object (pp. 216-226). London: Thames & Hudson.

7. Naur, P. (1992) Programming: A Human Activity. New York: ACM Press.

8. Sommerville, I. (2010) Software Engineering (9th Edition). Reading, Mass: Addison-Wesley.

9. Wegner, P. (1997) Why interaction is more powerful than algorithms. Communications of the ACM, 40 (5), 80-91.

10. Weinberg, G. (1971) The Psychology of Computer Programming. New York: Van Nostrand Reinhold.
<figure><img src="image_3.png" /></figure>
The APEX framework:
prototyping of ubiquitous environments based
on Petri nets

José Luís Silva¹,*, Óscar R. Ribeiro¹, João M. Fernandes¹,
José Creissac Campos¹, and Michael D. Harrison²

¹ Dep. Informática / CCTC, Universidade do Minho, Braga, Portugal
{jlsilva, orribeiro, jmf, jose.campos}@di.uminho.pt

² Newcastle University, United Kingdom
michael.harrison@ncl.ac.uk

**Abstract.** The user experience of ubiquitous environments is a determining factor in their success. The characteristics of such systems must be explored as early as possible to anticipate potential user problems, and to reduce the cost of redesign. However, the development of early prototypes to be evaluated in the target environment can be disruptive to the ongoing system and therefore unacceptable. This paper reports on an ongoing effort to explore how model-based rapid prototyping of ubiquitous environments might be used to avoid actual deployment while still enabling users to interact with a representation of the system. The paper describes APEX, a framework that brings together an existing 3D Application Server with CPN Tools. APEX-based prototypes enable users to navigate a virtual world simulation of the envisaged ubiquitous environment. The APEX architecture and the proposed CPN-based modelling approach are described. An example illustrates their use.

# 1 Introduction

Ubiquitous computing poses new challenges for designers and developers of in-
teractive systems. Because these systems immerse their users, the effect they
have on the users' experience is an important element contributing to the suc-
cess of a design. Technology enhancement has the potential to have a profound
impact on a built environment transforming a sterile space into a place that is
in harmony with its purpose. The experience of checking into an airport can
be improved by providing information to travellers when and where they need
it. Frustrating delays could thereby be removed through the appropriate use of
personalised information. The experience of using a library could be improved
by providing personal and clear information about the location of the shelf in a
large library where the required book is located. Experience therefore becomes
an additional interactive characteristic of ubiquitous systems, to be explored in
addition to more traditional notions of usability.

* José Luís Silva is supported by Fundação para a Ciência e Tecnologia (FCT, Portugal) through PhD Grant SFRH/BD/41179/2007.
*Experience* is difficult to specify as a requirement that can be calculated and demonstrated of a system. It is difficult to measure and to obtain early feedback about whether a design will have the required effect. Currently, there are no techniques that can be used to analyse specifications against different notions of experience (for a discussion, see [9]). An important barrier is the difficulty of developing prototypes that could feasibly be used to explore issues of experience.

This paper limits attention to *ubiquitous environments* envisaged as enhancing physical environments. In the envisaged designs, “spaces” are augmented with sensors, public displays and personal devices. Of particular interest in these systems is the way that the user interacts with the environment, as a result of both explicit interaction with the system, and implicit interactions that arise through changes of *context*. Here *context* could include location, or the steps that have to be taken by a user to achieve some goal (for example check-in, baggage screening, passport control, boarding card scanning).

The paper describes how prototypes can be built to represent the interaction between users, devices and services, as users move within ubiquitous environments. To avoid unnecessary development cost, early designs are explored in this proposal through model-based prototypes explored within a virtual environment. The paper describes a prototyping framework (APEX) that uses Coloured Petri Net (CPN) [11] models. APEX binds a CPN model to a 3D application server (OpenSimulator<fnref n="3" />).

The Petri nets modelling language, being an expressive and graphically informative notation, allows the description of the envisaged design. OpenSimulator provides support for exploring the design based on the Petri net description. Their integration thus allows rapid prototyping of ubiquitous environments, enabling users to navigate a virtual world simulation of the environment to evaluate usability issues, including user experience.

This paper builds on [18]. There, the early concept of the APEX framework was discussed, and some initial results presented. Since then, the framework has been developed, and the modelling approach fully revised. The new models present a number of benefits, including better scalability and support for heterogeneity. The current paper describes the APEX architecture, the new modelling approach, and provides modelling guidelines for developing prototypes.

The structure of the paper is as follows. Section 2 discusses related literature and the goals of the project. Section 3 describes the architecture of APEX. Use of the framework is illustrated by means of a smart library which senses the presence of users, and guides them to the shelves where their required books are located. Section 4 describes how the example is modelled. Section 5 describes usage of the framework. Section 6 presents conclusions and future work.

## 2 Related literature and goals

Despite considerable advances in the development of ubiquitous systems, there continues to be a tendency (see [5] for a concise overview) for the development

<footnote n="3">³ http://opensimulator.org (last accessed June 14, 2010)</footnote>
and evaluation of ubiquitous systems to be focussed on experimental systems, usually prototype device designs within partial systems. The issue of how to evaluate whole systems in real contexts continues to be a concern, see [2] for a useful discussion of this contrast. Another important aspect of evaluation is how to explore the user experience that a designed system creates. In this respect there is a substantial literature taken from design disciplines, see for example [4]. In design, for example, a typical approach is to use non-functional (for example, clay) prototypes as objects which potential users are asked to carry around in the contexts where the actual system is to be used in order to obtain information about how the proposed design might be experienced. One particularity of the type of systems of interest is that the system is woven into the context, making it harder to prototype.

APEX is designed to satisfy three requirements. The first is that it should enable the rapid development of both prototypes and target systems. While there are several existing platforms for ubiquitous computing ([3, 8, 10] are examples), a software tool is required that facilitates the development of prototypes, while simultaneously providing the hooks for the target system.

The second requirement is that a 3D environment can be used to construct simulations that can be explored realistically by users. 3D Application Servers, such as SecondLife™<fnref n="4" /> or OpenSimulator, provide a fast track to developing virtual worlds. OpenSimulator, in particular, has the advantage of being open source, which means that the backend can be programmed allowing configurability and extensibility.

Systems such as Topiary [13] enable users to explore prototypes of context-aware application in real world settings. They resort to Wizard of Oz techniques to avoid the actual deployment of sensors. They are targeted to the prototyping of applications running on user devices, and do not support the *enhancement* of the physical space. A different class of systems, such as 3DSim [17], UbiWorld [6] or the work of O'Neill et al. [16], have similar visions to ours (developing simulations of the actual environments).

The third requirement is an approach to modelling ubiquitous computing. While 3DSim and UbiWorld envisage the use of programming languages to build the prototypes, we are interested in creating them from models of envisaged systems. A benefit of this approach is the integration of the modelling approach with analytical approaches, to provide leverage on properties of ubiquitous environments that are relevant to their use.

Petri nets constitute an expressive and graphically informative modelling language that has been used to describe virtual environments. Previous modelling approaches based on Petri nets include the use of: Hybrid high-level Nets (HyNets) [14], Flownets [19], Interactive Cooperative Objects (ICO) [15], and Coloured Petri Nets (CPN) [11].

CPN modelling and analysis is supported by CPN Tools, enabling analysis either by simulation (similar to program execution) or by more formal analysis (state space analysis and invariant analysis). Simulation can be used to animate

<footnote n="4">⁴ http://secondlife.com (last accessed June 14, 2010)</footnote>
the models. State space analysis can be used to check standard properties, such as reachability, boundedness, liveness properties and fairness, as well as specific properties defined using the associated programming language (CPN ML language [12]).

In summary then, given the objectives set forth for APEX, CPN was chosen because: (i) it allows rapid development of prototypes, much faster than equivalent conventional approaches using C#; (ii) it allows analysis of properties of the model (via CPN Tools); (iii) the animation capabilities of CPN Tools allow control of the virtual world simulations directly from the models (hence, the behaviour modelled is exactly what is executed — this improves on current approaches in that in these approaches when a simulation needs to be programmed, what is executed does not necessarily reflect the models and specifications produced in an earlier development stage).

While several approaches aiming at ubiquitous computing prototyping were identified above, they are mostly focused on helping ubiquitous system designers to identify unwanted behaviour in their system, and to support informed decision making in an iterative design cycle. APEX is more focused on the *experience* users will have of the design, and in the use of tools to enable analysis.

The above mentioned approach of O'Neill et al. [16] is the most similar to ours, using models and a 3D simulation for the prototyping of ubiquitous environments. In their case a games engine is used. We believe the use of a 3D application server (OpenSimulator) has some advantages compared with a games engine. It supports the creation of virtual environments in real time using world building tools, and it is easily extendable by the loading of modules. In the case of the games' engine, the environment must be previously fully created using a map editor. Using a 3D application server means the approach is flexible. A variety of clients, customizable in appearance, can access the virtual world on multiple protocols at the same time, and in world application development using a number of different languages is also possible.

## 3 The APEX framework

The overall architectural view of the APEX framework is presented in Figure 1. Three main components are identified:

- a *virtual environment component*, responsible for managing the physical appearance and layout of the prototype, including managing the 3D simulation and the construction of the virtual environment;

- a *behavioural component*, responsible for managing the behaviour of the prototype, including the description, analysis and validation of the virtual environment's behaviour;

- a *communication/execution component*, responsible for the data exchange among all components and for the execution of the simulation.

OpenSimulator enables the interactive creation of virtual environments. It provides a sufficiently rich *texture* to enable users to visualise the physical char-
<figure><img src="image_4.png" /><figcaption>Fig. 1. Logical architecture of the APEX framework</figcaption></figure>

acteristics of the real system. A rich palette of features provides for easy ob-
ject/environment creation and manipulation. These objects, together with the
insertion and manipulation of textures, lighting, animation and sounds also pro-
vided, enable a simulation which can create a realistic visualisation of the pro-
posed real system. Pre-defined environments and devices can be used in this
creation process.

To create a prototype, besides creating the virtual environment, the developer needs to extend the CPN base model provided. APEX uses CPN Tools to model the behaviour of the virtual environment. Models of each type of dynamic object/device in the environment (e.g., sensors, displays, personal devices) need to be inserted into the global model of the environment. Adequate models must either be available or must be created using CPN Tools. Section 4.2 will provide a more detailed description of how that can be done.

Once the CPN model and the environment are created a component of the
framework binds them together. To achieve this, transitions in the CPN link the
behaviour described by the models to the respective objects in the environment.

Several users can be connected to the simulation using different viewpoints onto the OpenSimulator server. Users can navigate and interact with the virtual world simulation of the envisaged ubiquitous environment, enabling the evaluation of usability and experience issues with the proposed design.

**3.1 Behavioural component**

This component is responsible for driving the simulation using the information
from the model, and to send relevant data to the virtual environment. It contains
the CPN tools, which use CPN models to describe the behaviour of the virtual
environment in response to user actions and context changes.

A generic CPN base model is provided from which virtual environment mod-
els can be derived. The aim in developing this base model was to develop a
generic style of CPN relevant to the modelling of virtual environments, includ-
ing models that can be instantiated to the physical space in which the system
is to be defined to operate. The model consists of: (a) a module to initialise the simulation, and to establish the connection between the CPN model, as represented by CPN Tools, and OpenSimulator; (b) a module that receives user data (for example user identity and position) from OpenSimulator when a user moves and uses it to update appropriate tokens; (c) modules describing the behaviour of each device in the system. An example is presented in Section 4.

## 3.2 Virtual environment component

This component sends information about the simulation (e.g. user position) to the behavioural component which takes a *decision* and sends indications to reflect these changes in the simulation. It contains the OpenSimulator server and viewers for each client who connect to it.

The OpenSimulator server is responsible for maintaining the virtual environment information available to viewers. The features of the 3D simulation include location, the viewing aspect and the physics of each of the objects in the environment. Pre-defined environments and objects can be saved/loaded in/from Opensim Archive files (OAR). All the different entities (object, terrain, textures, etc.) are packaged in these files in the format used by Opensimulator to keep data within an archive. The server enables the connection of several users from, possibly, different locations to the same virtual environment via the web through appropriate viewers.

Viewers interact with the server and are used to define features of the 3D simulation presented to users, and to allow users to navigate and interact within the simulated environment. Interaction is achieved both explicitly by a user using (virtual) devices, and implicitly through changes of context. Possible viewers include the Hippo OpenSim Viewer<fnref n="5" /> or the Linden Lab's Second Life viewer<fnref n="6" />. However, a number of alternative compatible viewers exist<fnref n="7" />. Note that, currently, some of these alternative viewers only enable the environment exploration without providing any modelling tool.

The behaviour described in the previous section is linked to the objects which are identified by unique names. For instance, to open a gate in the simulation, the CPN model of the gates must indicate in its open transition code the identifier of the gate to open. Objects identifiers are easily accessible through the properties panel provided by the viewer and associated to each object of the environment.

## 3.3 Communication/execution component

This component is a DLL (dynamic-link library) responsible for loading the simulated ubiquitous environment into the OpenSimulator server, and for using the CPN models to drive it. It is positioned between the two other components managing the exchange of information between them.

<footnote n="5">⁵ http://mjm-labs.com/viewer/ (last accessed June 14, 2010)</footnote>

<footnote n="6">⁶ http://secondlife.com/support/downloads (last accessed June 14, 2010)</footnote>

<footnote n="7">⁷ http://opensimulator.org/wiki/Connecting (last accessed June 14, 2010)</footnote>
Communication in the CPN models is achieved through Comms/CPN [7], a CPN ML library for connecting between CPN Tools and external processes, provided with the CPN Tools. The BRITNeY Suite [20] also enables the communication between CPN models and a Java-based animation package. Comms/CPN is more adequate and simple to use for our case. Unlike Comms/CPN the BRITNeY Suite has a more general purpose, providing more features besides the communication package, which make it more complex to use.

In order to use Comms/CPN a module must be loaded into the external process. Java and C modules are available with the distribution. However, OpenSimulator modules (DLLs) are developed in C#. No alternatives were found for this communication so a new C#/CPN communication package has been developed. With this development the communication of the CPN models, using the Comms/CPN functions, and C# processes becomes possible.

The developed module sends information to CPN Tools when changes in the environment happen, and is responsible for changing the environment in response to data sent by CPN Tools. Additionally, it handles the loading/saving of OpenSimulator objects/environments and the execution of commands invoked by the user in the viewer. When inserted in the OpenSimulator server location, this DLL is automatically loaded by the OpenSimulator. After the establishment of the communication between the CPN model and the simulator, by the evocation of a function in the CPN model (explained in the next section), the APEX is ready to use.

# 4 Modelling with CPNs (The example)

As previously stated, a generic CPN modelling approach was developed to enable the easy creation of new ubiquitous systems prototypes. In this section the modules of this approach are described. Figure 2 presents the setup model. As will be discussed, this model needs small modifications only when being adapted to different applications. The model in figure 3 deals with user position and is generic. The developer then needs to develop a module for each device type present in the ubiquitous system. Figure 4 presents the module for a specific type of object present in the example used (a gate). How these modules are created will be described in section 4.5

## 4.1 The example

The example used to illustrate the system is a smart library. Books are identified by RFID tags and are stored on bookshelves. Screens are used to provide information to library users. A registered library user is allowed entry/exit via gates. When a registered user arrives at the entry gate, a screen displays which books have been requested by the user (e.g., earlier via a web interface) and opens the entry gate. The system guides the user to the required books through the use of sensors that recognise the user's position in real-time. As the user approaches the book's location a light with a specific colour is turned on. Hence several users
looking for books in nearby locations can distinguish their own request. When
the book is removed, the light on the book is turned off. As the user returns to
the exit gate a personalised list of requested and returned books is displayed on
a screen by the gate which is opened so that the user can leave.

## 4.2 Modelling approach

There are a number of styles of specification that can be achieved using CPN.
These styles vary according to the extent to which the semantics of the under-
lying objects are made explicit in the structure of the CPN specification, or
encoded into the tokens. The following two extremes are possible:

- placing all the semantics in the tokens, in other words, minimising the num-
  ber of places in the net;

- using places to characterize each different relevant situation (user action, context change, etc.), thereby adding transitions that explicitly describe aspects of the semantics of the objects.

A small example is presented to clarify these two approaches. Suppose a
device which can be in two different states (**on** and **off**) is to be modelled.
Following the two approaches above, two different results will be reached. In the
first, the model will consist of only one place, and one transition from and to
this place. The place will hold tokens with a semantics which can represent all
the different states of the device. The state of the device will be encoded as an
attribute (a colour) of the token representing the device. The transition will be
responsible for changing the colour of the token, reflecting the new state of the
device. In this situation all the meaning is in the value of the tokens.

Following the second approach, the model will be represented by two places
each representing a possible state of the device, and by transitions between them
(two in this case). No semantics will be carried by the token, all the meaning will
be represented by the structure of the model. The state of the device is known
by looking to the position of the token, i.e. at the place which holds the token.

In APEX, a mixed approach is used where the states of the dynamic objects
(open, closed, etc.) are modelled as places and user actions and context changes
modelled as transitions. Each device and user is represented in the CPN model
as a token in the respective place. Each of these tokens has an identifier which
is used as the identifier of the objects present in the simulation.

The users and object features (e.g. identifier, position) are modelled as at-
tributes in their respective tokens. These values are used by CPN ML functions
together with instructions (e.g. *open*, *close*) to indicate changes that must be
reflected in OpenSimulator. Section 4.5 will provide a description of how this is
done. The guards on the transitions as well as the functions associated with tran-
sitions are responsible for part of the behaviour of the system. Both of these are
modelled in the CPN ML language, so this behaviour is modelled functionally.

This combination gives more expressiveness to the ubiquitous systems mod-
elling while avoiding clutter in the CPN specification. In the next sub sections,
the approach will be illustrated using the example.
<figure><img src="image_5.png" /><figcaption>Fig. 2. The CPN module to setup the library simulation.</figcaption></figure>

## 4.3 Setting up the simulation

The initial conditions of the simulation are defined in the CPN module shown in Figure 2. Firing the “initialise simulation” transition sets the initial configuration of the simulation, and executes the associated CPN ML code. For example, “acceptConnection(connName,9002)” is a function of the Comms/CPN library used to establish the connection between CPN Tools and OpenSimulator.

In this case the configuration includes three places: “users”, “gates” and “bookshelves”. Fusion tags (inset into the lower left corner of the places) enable instances of these places to appear in other parts of the CPN model. Hence, these places are called *fusion places*. The utilization of these places will be described in section 4.5.

Annotations at the bottom right side of the places indicate the type of token each place can hold. Place “users” holds “USER” tokens representing information about users in the virtual environments. This particular place is mandatory, since whatever the model the handling of users must be supported. The remaining places (“gates” and “bookshelves”) hold tokens representing devices. These places are system dependent and will vary for each prototype. The colour (structure) of the tokens which these places can hold is defined in CPN Tools, and characterises the information held in the model for each type of device.

Besides establishing the connection between the CPN tools and OpenSimulator, and initializing user and device places, the “initialise simulation” uses two places to control the execution of the CPN model: “init” to limit execution of the transition to one occurrence, and “run” to inform other CPN modules that the simulation is running.

## 4.4 Reading users' positions

Figure 3 presents the CPN module that collects users' data from the OpenSimulator. Transition “read user id” reads a user identifier sent by the OpenSimulator server (c.f., “receiveString()” function on the code block associated with the transition). A token with the value of the read user identifier is in-
<figure><img src="image_6.png" /><figcaption>Fig. 3. The CPN module for acquiring users' data.</figcaption></figure>

troduced in the "read user ids" place (c.f., "output(idRead)"). This is used to read the new position by means of the transition "read and update user position", which also updates the relevant user token (taken from the user's fusion place). The new coordinates *x* and *y* are read in the *action* part of the transition using the function "receiveInteger()" and a pair of coordinates (*p*) is produced. This pair is then used to update the user position through the "updateUserPosition" function. The expression "isThisUser(u,uId)", in the guard of this transition, guarantees that the user token which is updated corresponds to the previously read identifier. In this model the number of users remains constant during each simulation session. To add more users to the system one must add the corresponding tokens in the place "users". The automatic addition and deletion of users at runtime, in accord with the users connected to the simulation, is planned. This will be achieved via the addition of new models to generate user tokens, and by enhancements to the C# module.

CPN modules for reading the user's position, and for managing devices' be-
haviour execute concurrently. Precedence of devices' transitions over data ac-
quisition transitions is guaranteed through the guard "not (hadASignificant-
Movement(u))" on the transition "read and update user position". Move-
ment of a user is significant (for a device) when the new position is "near" the
device. Hence, if a user is near a device, no new data will be acquired until the
device has processed the current data.
<figure><img src="image_7.png" /><figcaption>Fig. 4. The CPN module for an entry gate device.</figcaption></figure>

## 4.5 Modelling the devices of the system

Each device type in the ubiquitous environment simulation needs a corresponding CPN module describing its behaviour. It is envisaged that a library of models will be made available for supported devices. When new (unsupported) devices are to be used, a new model must be developed and added to the library. This section explains the process through the example of the entry gate.

Device behaviour is modelled through a combination of fusion places, normal places, transitions, functions (described in the CPN ML language) and conditions. State transitions play an important role in this process since it is through them that the connection between the model and the simulation is accomplished, via the associated CPN ML functions. These functions are also responsible for describing functional behaviour not structurally expressed by the net.

Fusion places are the basis for the creation of these behavioral modules. They establish the link between the models of the devices and the setup model presented in figure 2. The device model for the entry gate model is presented in Figure 4. In this example the users and gates fusion places hold the (user and gate) tokens needed to model the behaviour.
The entry gate is equipped with a sensor to capture a user approaching it. Transition “show info and open gate” represents the actions of the entry gate. It displays requested books on the screen and opens the entry gate. Functions “sendShowInfo” and “sendOpenGate” are responsible for these actions sending relevant instructions to OpenSimulator. These actions occur when the gate’s sensor detects a registered user arriving at the entry gate (modelled by “isArrivingToGateArea(u,g)” evaluating to true). When the gate is open and another registered user enters the gate area the transition “add a user” occurs and this user is included in the set of users that are near the gate. This set of users is represented in the tokens held by the place “open gates”. As already stated, each place has an associated token type which it holds. In this case the type of this place is “USERIDsxGATE”. It means that each token is a product of a set of user IDs (“USERIDs”) and the gate (“GATE”) which they are near.

When the transition “show default and close gate” is taken, default information is displayed on the screen and the gate is closed. For this to happen a user must have moved away from the gate, and there should be no more users near it. If other users are near the gate, the transition “remove a user and update info” removes the designated user from the list of users who are near the gate (function “removeUser”) and, if that user’s information was being displayed, the information currently on the screen is changed to one of the other user’s (function “sendShowInfo”).

As explained, the different CPN models are connected via *fusion places*, enabling token flow between them (e.g. the models in figures 3 and 4 are connected to the setup model via the “users”, “gates” and “run” *fusion places*). Put together they form the model of the envisaged ubiquitous system.

The focus of this paper had been the architecture of the framework, and the CPN-based modelling. Of course, a virtual environment to match the model must also be developed. This is done through the virtual world viewer. Once both models and virtual world simulations for all devices are in place, animation of the envisaged ubiquitous system can start.

# 5 Support for design

As stated in Section 1, APEX supports both the design and the analysis of ubiquitous systems. The developer creates the CPN model, as illustrated in Section 4. Depending on the new types of devices that are used the developer is required to modify a small piece of the Communication/Execution C# module responsible for reflecting the changes in the simulation. The code responds to changes in objects of the environment consistent with the state of the CPN model. As an example, Figure 5 is a code snippet that searches for objects where changes are directed to occur by the CPN model and makes the changes. In this snippet an *open* or *close* action is received and the position of the gate is changed accordingly.

A typical runtime configuration of the framework (see figure 6) will involve deploying the OpenSimulator server, CPN tools, and the Communica-
```csharp
foreach (KeyPair<Scene, List<SceneObjectGroup>> kvp in HelloWorldModulescene_prims)
{
    foreach (SceneObjectGroup sog in kvp.Value)
    {
        //Object: gate
        if (sog.Name == "gate")
        {
            if (action == "open") //Open the gate
                sog AbsolutePosition = new Vector3(130.7f, 130.9f, 25.9f)
            else //Close the gate
                if (action == "close")
                    sog AbsolutePosition = new Vector3(128.6f, 130.9f, 25.9f)
        }
        //Object: screen
        ...
    }
}
```

<figure><img src="image_8.png" /><figcaption>Fig. 6. Physical architecture of the APEX framework</figcaption></figure>

tion/Execution module on a server. Once the CPN model is loaded, the server is ready to allow free exploration and interaction with the virtual environment. At this point, exploration and interaction with the virtual environment is possible. Currently this is achieved by means of viewers deployed on client machines. It is envisaged that higher fidelity prototypes will be possible, for example, using a CAVE system.

Using these prototypes, it becomes possible to test different design alternatives with real users, without the cost of developing the actual system. As an illustration, figure 7 shows a user collecting a book. As the (registered) user approaches the gate (step a) the gate opens and the user is able to enter the library (step b). Once the user is close to the book, the light on the book is turned on so that the user can quickly identify it (step c).

Validating the usefulness of these prototypes in assessing users' experience of the envisaged systems will be the subject of a next phase in the project. However, the literature on virtual reality for purposes such as education, training
<figure><img src="image_9.png" /><figcaption>**Fig. 7.** Viewer interface - user common path</figcaption></figure>

or medical treatment, contains good indications these systems provide for a rich enough experience to allow relevant results to be reached (e.g., see [1] for some interesting papers on the applicability of virtual reality to behavioural sciences).

In addition to exploring the environment, it is also possible to use the viewer to manipulate it, load objects into the environment and to save and clear the environment. This is achieved in the viewer by an avatar “shouting” commands: *load-oar file*, *save-oar file* and *clear*.

Besides exploration of the prototype, analysis of the models can also be considered. Using the State Space tool, provided with the CPN Tools, properties can be checked in the model. For instance, reachability properties (e.g. all the states are reachable, a state is reachable from another one) can be expressed using functions provided by the State Space tool for this effect (e.g. AllReachable(), Reachable(node, node)). In the example, given specific assumptions about user behaviour, captured by adding an automated avatar to replace free user interaction, the model can be used to check properties such as that the required book will always be reached, collected and taken out of the library.

# 6 Conclusions and Future work

The user experience of ubiquitous environments is a determining factor in their success. Enabling early exploration of the characteristics of such systems will help anticipate potential user problems and reduce the cost of redesign. However, the deployment of prototypes in the target environment is, in many cases, infeasible. This happens both because of the cost of deploying such prototypes, and because doing it can be disruptive to the ongoing system. Alternatives must be sought that capture the experience of being immersed within the proposed ubiquitous system, without the cost of actually fielding it.

This paper described one such alternative. A simulation-based prototyping framework for ubiquitous computing systems. The framework brings together the expressive and analytic power of Petri nets, with the possibility of exploring a 3D virtual simulation of the modelled system. Petri nets constitute an ex-
pressive graphical notation. Development of the models and 3D environments is
accelerated by the use of the CPN base model, and pre-defined devices. By en-
abling potential users to explore the simulation of the system before deployment,
it becomes possible to have a low-cost approach to the prototyping problem.

Ongoing work on the development of the framework is addressing a number of
technical issues in order to better support developers and users. One immediate
aspect is the possibility of adding users to the simulation at runtime. In the
current version of APEX, the number of users must be set at the start of the
simulation run. This will be fixed in the next version of the framework. Another
goal is reducing the amount of information exchanged by CPN Tools and APEX
to a minimum. This is relevant both to prevent CPN Tools from running out
of resources, and because it is envisaged that simulations will be deployed via
the web. Connecting the simulation to user devices via bluetooth is also being
addressed. This will encourage a more immersive and realist usage experience
by allowing mixed reality. It also allows the possibility of moving progressively
as part of the design and implementation process from a simulated system to a
real system. Exploring the formal analysis of the models is also being considered.
This requires the development of simulated users (capturing assumptions about
user behaviour) to allow for a complete analysis. Hence, combining this feature
with the previous one, progress will be made towards a mixed economy of
simulated and actual components of a proposed design. This will also support
exploring how different levels of abstraction can be accomplished and supported.
For example, supporting and enabling the migration of devices at the physical
level via Bluetooth, at the virtual level as virtual devices in OpenSimulator, at
the model level as CPN models.

Further development of the framework will involve its evaluation with users
and developers. User evaluation concerns the fidelity of the results. Whether
prototype environments can be used effectively to enable users to experience
the design. Developer evaluation is concerned with the approach's agility. It is
concerned with the ease with which accurate prototypes can be developed for
ubiquitous environments.

References

[1] CyberPsychology & Behavior. Volume 6, Number 3/4 (2003)

[2] Abowd, G., Hayes, G., Iachello, G., Kientz, J., Patel, S., Stevens, M., Truong, K.: Prototypes and paratypes: designing mobile and ubiquitous computing applications. IEEE Pervasive Computing 4(4), 67–73 (2005)

[3] Braubach, L., Pokahr, A., Moldt, D., Bartelt, A., Lamersdorf, W.: Tool-supported interpreter-based user interface architecture for ubiquitous computing. In: Interactive Systems. Lecture Notes in Computer Science, vol. 2545, pp. 89–103. Springer-Verlag (2002)

[4] Buchenau, M., Suri, J.: Experience prototyping. In: Proceedings Designing Interactive Systems (DIS’00). pp. 424–433. ACM Press (2000)

[5] Davies, N., Landay, J., Hudson, S., Schmidt, A.: Rapid prototyping for ubiquitous computing — guest editors’ introduction. IEEE Pervasive Computing 4(4), 15–17 (2005)
[6] Disz, T., Papka, M., Stevens, R.: UbiWorld: an environment integrating virtual reality, supercomputing, and design. In: Proceedings of the Heterogeneous Computing Workshop. pp. 46–59 (April 1997)

[7] Gallasch, G., Kristensen, L.: Comms/CPN: A communication infrastructure for external communication with design/CPN. In: Jensen, K. (ed.) 3rd Workshop and Tutorial on Practical Use of Coloured Petri Nets and the CPN Tools (CPN’01). pp. 75–90. DAIMI PB-554, Aarhus University (2001)

[8] Garlan, D., Siewiorek, D., Smailagic, A., Steenkiste, P.: Project Aura: toward distraction-free pervasive computing. IEEE Pervasive Computing pp. 22–31 (April–June 2002)

[9] Harrison, M., Campos, J., Doherty, G., Loer, K.: Connecting rigorous system analysis to experience centred design. In: Law, E., Hvannberg, E., Cockton, G. (eds.) Maturing Usability: Quality in Software, Interaction and Value, pp. 56–74. Human Computer Interaction Series, Springer-Verlag (2008)

[10] Harter, A., Hopper, A., Steggles, P., Ward, A., Webster, P.: The anatomy of a context-aware application. Wireless Networks 1, 1–16 (2001)

[11] Jensen, K., Kristensen, L., Wells, L.: Coloured petri nets and CPN tools for modelling and validation of concurrent systems. International Journal on Software Tools for Technology Transfer (STTT) 9(3-4), 213–254 (2007)

[12] Jensen, K., Kristensen, L.M.: Coloured Petri Nets – Modelling and Validation of Concurrent Systems. Springer (2009)

[13] Li, Y., Hong, J., Landay, J.: Topiary: a tool for prototyping location-enhanced applications. In: UIST ’04: Proceedings of the 17th annual ACM symposium on User interface software and technology. pp. 217–226. ACM (2004)

[14] Massink, M., Duke, D., Smith, S.: Towards hybrid interface specification for virtual environments. In: Duke, D., Puerta, A. (eds.) Design, Specification and Verification of Interactive Systems ’99. pp. 30–51. Springer-Verlag (1999)

[15] Navarre, D., Palanque, P., Bastide, R., Schyn, A., Winckler, M., Nedel, L., Freitas, C.: A formal description of multimodal interaction techniques for immersive virtual reality applications. In: INTERACT 2005. Lecture Notes in Computer Science, vol. 3585, pp. 170–183. Springer-Verlag (2005)

[16] O'Neill, E., Lewis, D., Conlan, O.: A simulation-based approach to highly iterative prototyping of ubiquitous computing systems. In: 2nd International Conference on Simulation Tools and Techniques. pp. 1–10. ICST (2009)

[17] Shirehjini, A.A.N., Klar, F.: 3DSim: rapid prototyping ambient intelligence. In: Proceedings of the 2005 joint conference on Smart objects and ambient intelligence: innovative context-aware services: usages and technologies (2005)

[18] Silva, J., Campos, J., Harrison, M.: An infrastructure for experience centred agile prototyping of ambient intelligence. In: Calvary, G., Graham, T., Gray, P. (eds.) Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. pp. 79–84. ACM Press (2009)

[19] Smith, S., Duke, D., Massink, M.: The hybrid world of virtual environments. Computer Graphics Forum 18(3), C287–C307 (1999)

[20] Westergaard, M., Lassen, K.B.: The britney suite animation tool. In: Donatelli, S., Thiagarajan, P.S. (eds.) Petri Nets and Other Models of Concurrency - ICATPN 2006. Lecture Notes in Computer Science, vol. 4024, pp. 431–440. Springer (2006)
# Model-based Design and Implementation of Interactive Spaces for Information Interaction

Hans-Christian Jetter, Jens Gerken, Michael Zöllner, and Harald Reiterer

AG Mensch-Computer-Interaktion, Universität Konstanz,
Universitätsstraße 10, 78457 Konstanz, Germany
{hans-christian.jetter,michael.zoellner,
jens.gerken,harald.reiterer}@uni-konstanz.de
http://hci.uni-konstanz.de

**Abstract.** Interactive spaces with multiple networked devices and in-teractive surfaces are an effective means to support multi-user collocated collaboration. In these spaces, surfaces like tablet PCs, tabletops, or dis-play walls can be combined to allow users to interact naturally with their personal or shared information, e.g. during presentation, discussion, or annotation. However, designing and implementing such interactive spaces is a challenging task due to the lack of appropriate interaction abstrac-tions and the shortcomings of current user interface toolkits. We believe that these challenges can be addressed by revisiting model-based design techniques for object-oriented user interfaces (OUI). We discuss the po-tential of OOUIs for the design of interactive spaces and introduce our own object-oriented design and implementation approach. Furthermore we introduce the ZOIL (Zoomable Object-Oriented Information Land-scape) paradigm that we have used as an experimental testbed. While our approach does not provide automated model-driven procedures to create user interfaces without human intervention, we illustrate how it provides efficient support throughout design and implementation. We conclude with the results from a case study in which we collected empirical data on the utility and ease of use of our approach.

**Keywords:** Interactive Spaces, Information Interaction, Zoomable User Interfaces, Model-based Design.

## 1 Introduction

Recent work in Human-Computer Interaction (HCI) suggests the use of physical work environments with multiple interactive surfaces (e.g. multi-touch tabletops or walls) for the collocated collaboration of multiple users. These “interactive spaces” are often used to support groups during the collaborative management, presentation, and discussion of information items, e.g. in science, design, and engineering [23, 7, 17]. Following the Weiserian vision of ubiquitous computing, a fundamental requirement for such interactive spaces is a “natural” style of human-computer interaction where computing interfaces ideally become invisib-le and unobtrusive. They vanish into the background of our familiar non-digital
reality. Therefore the essential operations of our information interaction such as viewing, editing, (re)locating, sharing, and annotating information items should be provided by natural or “reality-based” interfaces. Following Jacob et al.’s notion of reality-based interaction, such interfaces “draw strength by building on users pre-existing knowledge of the everyday, non-digital world to a much greater extent than before.” They attempt to make computer interaction more like interacting with the real, non-digital world by employing themes of reality such as users understanding of physical objects or their body and social skills. Fig. 1 shows an example of an interactive space and different reality-based interaction techniques that can provide a more natural and fluid user experience that is ideally not impaired by obtrusive computer user interfaces and technology-induced barriers between them.

<figure><img src="image_10.png" /><figcaption>**Fig. 1.** A ZOIL-based interactive space as realized in our lab (top). Natural interaction styles used in our ZOIL case studies, e.g. tangibles and digital pens (bottom).</figcaption></figure>
To this day, designing and implementing reality-based and tangible user in-
terfaces (UI) for interactive spaces is a challenging task. As discussed by Shaer
and Jacob, typical challenges are the lack of appropriate interaction abstractions,
the shortcomings of current user interface software tools to address continuous
and parallel interactions, as well as the excessive effort required to integrate
novel input and output technologies [22]. We believe that these challenges can
be addressed by viewing interaction through the lens of object-orientation. We
suggest to revisit the user interface modeling and design techniques for object-
oriented user interfaces (OOUI) from the 1990's that have widely fallen into
oblivion and to apply them on today's novel post-WIMP (post-"windows icons
menus pointer") technologies and user interface toolkits. In this paper, we make
three contributions to this field of research: In chapter 2, we discuss why we
believe that this step into the past era of OOUIs has great potential for the
design of future computing environments and why this is especially true when
considering collaborative interactive spaces for reality-based information inter-
action. In chapter 3, we introduce the ZOIL (Zoomable Object-Oriented Infor-
mation Landscape) paradigm that we have used as an experimental testbed for
our model-based design and implementation approach. In chapter 4, we illustrate
and discuss our approach for modeling OOUIs in detail. While our approach does
not provide automated model-driven procedures to create user interfaces with-
out human intervention, we illustrate how it can provide efficient model-based
support throughout the design and implementation and we present results from
a case study in which we collected empirical data on the utility and ease of use
of our OOUI approach from designers and developers.

## 2 Objects in Collaborative Information Interaction

There is a variety of high-level frameworks in HCI for modeling information in-
teraction, e.g. Blandford and Attfield’s “information journey” [4] or the GEMS
model from Salminen et al. [15]. Typically these models consider information
interaction as a task-oriented series of phases of higher level activities that are
separated in time, e.g. *recognizing an information need, acquiring information,*
*interpreting information, and using interpretation*. Such generic frameworks can
be used as a starting point for interaction design: During a *top-down* design
process, these generic high-level activities can be contextualized for the targeted
application domain and can be hierarchically decomposed into domain-specific
lower level task models (e.g. essential use cases or scenarios). These are used to
define the abstract user interface architecture (e.g. the navigation map) and to
later flesh out the details of the concrete visual design of individual pages or di-
alogs. Such a task-oriented top-down design process (e.g. usage-centered design
[6]) creates interfaces that resemble virtual pathways to guide users through all
the stages, information resources, and interaction contexts that are necessary
for completing the tasks from the application domain. These page flows or se-
ries of dialogs define the virtual routes that users can take when working with
the system. Under the influence of the page-oriented World Wide Web, inter-
action designers have become very experienced in designing interfaces as such task-oriented stepwise conversations between a single user and a system that move along predefined paths. They achieve great usability for domains with a finite number of clearly defined tasks or business processes (e.g. in e-commerce). However, we believe that in the post-WIMP era such purely task-oriented thinking during design and implementation cannot leverage the true power of today's novel ways of natural and collaborative interaction.

## 2.1 Task-Orientation vs. Object-Orientation

In the case of collaborative information interaction in post-WIMP environments like in Fig. 1, designers have to consider interaction not only as a task-oriented sequential process supported by a single interface and its hard-coded functionality. In such settings, information interaction becomes a distributed, concurrent, and sometimes seemingly chaotic activity that does not follow simple task models. Instead, the users' actions are situated in a constantly changing social and technological setting, in which multiple users at multiple points of action simultaneously pick up, use, manipulate, recombine, create, and destroy virtual information objects without following clearly defined processes that terminate at clearly defined goals. Furthermore, such post-WIMP environments with multi-touch or tangible user interfaces) also afford more natural interaction styles. Instead of clicking hyperlinks or widgets as an intermediary language to sequentially converse with a system about intended actions, users want to continuously touch, grab, and manipulate physical or virtual objects from the application domain. Ideally the application domain itself becomes directly user-accessible and user tasks are carried out by directly manipulating the objects representing it. Thus the user interface changes its nature from being a task-oriented intermediary language medium based on widgets into a computer-mediated world of cooperating visual and tangible objects that provide users with more means for flexibility, improvisation, and establishing individual working styles.

The challenge of designing and programming interfaces that are entirely based on the direct manipulation of cooperating objects instead of sequential conversations is not new. It is similar to the challenge that designers were facing during the advent of graphical user interfaces and direct manipulation in the 1980s [22]. At that time, Hutchins et al. referred to this new kind of direct manipulation interfaces as “model-world interfaces” as opposed to traditional interfaces which have been designed with a conversation metaphor of human-computer interaction in mind [10]. Model-world interfaces provide a coherent and consistent overall representation of the application domain in which the user can freely navigate and directly act on domain objects using a series of low-level direct manipulations that in sum constitute the intended high-level tasks and activities. Essentially, the design challenges we face now in the design of interactive spaces are the same: How can we break down an application domain and its higher level tasks into cooperating visual and tangible objects inside an interactive space, in which higher level tasks can be carried out in natural ways by lower level direct manipulations of objects?
## 2.2 Revisiting Object-Oriented User Interfaces (OUI)

In the 1990s, IBM introduced the term *Object-Oriented User Interfaces* (OUI) to describe a new kind of direct manipulation model-world interfaces: “An object-oriented user interface focuses the user on objects - the “things” people use to accomplish their work. Users see and manipulate object representations of their information. Each different kind of object supports actions appropriate for the information it represents” [21]. At that time, OUIs were considered as more usable due to the closer match between the application domain and its virtual counterpart on the screen. Furthermore, unlike application-oriented user interfaces, OUIs provided greater flexibility and consistency following a “flexible structure-by object” instead of a “rigid structure-by function” [16]. Today, this makes OUIs particularly interesting for post-WIMP designs that are intended to better support the unpredictable and ill-defined needs and actions of situated users which cannot be anticipated by the task models of the design phase.

During OUI design it is important to avoid unnecessary realism in interface metaphors or an unintelligible plethora of different object types and behaviors. To achieve this, OUI designers employ rigid object-oriented mechanisms such as *inheritance*, *generalization*, and *polymorphism* to analyze and model the essential characteristics of the application domain. Thereby they view the domain through the lens of object-orientation from a user's perspective. Using these mechanisms, the user-perceived similarities and differences between domain object types are modeled in common base classes or subclasses. “Interactions should be consistent across objects of the same class; where possible, operations should be polymorphic - applicable to different object types. This reduces the number of interaction behaviors and simplifies the interface” [5]. This way the modeled class hierarchy can integrate very different types of domain objects into a single model while preserving a maximum degree of consistency in interaction. This model is then used to design and implement an interface with consistent behavior, functionality, and appearance. If properly applied users experience a “logical” behavior throughout the entire OUI. Thus they can more easily apply their previous experiences to infer their strategies for handling novel tasks.

Although OUIs strongly influenced the design of the “desktop metaphor” in today’s operating systems, OUI design approaches have not been subject of intense scientific research. Most efforts only lasted until the late 1990s (e.g. [1, 2, 16, 5, 21]) and after that there has only been some OUI-related work in the context of Pawson’s radical *Naked Objects Pattern* which tries to eliminate the need for specific user interface design by making all code objects and data models directly user accessible [18]. In conclusion, we are aware of only two publications that have proposed entire OUI design methodologies: IBM’s comprehensive description of the OVID methodology in [21] and the brief description of Beck et al.’s TASK methodology in [2].

The OVID methodology (*Object, View, and Interaction Design*) for OUI design was intended to bridge user interface and software engineering by using the UML notation and modeling techniques of successful code design and combine these with user interface design and usability engineering. At the heart of
OVID is the *designer's model*, a conceptual model that includes “descriptions of the objects users will employ to perform their tasks, the properties of those objects, and the interrelationships between them” [21]. To identify the objects that users have to act on and that should be provided to them on the user interface, textual and formal notations of tasks (e.g. use case diagrams) can be used, so that “task analysis will reveal information about what the users do and which objects they work with”. Despite OVID's comprehensive treatment in [21], only high level descriptions of iterative design and prototyping are provided and many of the necessary steps, rules, or tools remain unclear.

Before OVID, Beck et al. introduced the TASK methodology for integrating OO analysis into graphical user interface design for desktop systems [2]. During TASK's analysis activity, a task model and an initial object-oriented *object model* is built, which is then refined to an object-oriented *application specification*. This specification is used as a *conceptual user interface model* during user interface design and the views, dialogs, and the actual screen representations of conceptual objects are derived from it. The successful application of TASK and its supporting tools is mentioned for the design of insurance and production planning systems. However, the detailed tools, rules, and the amount of human intervention for translating the *conceptual user interface model* into concrete user interface design and its implementation are not revealed in detail.

## 3 Exploring OOUI approaches using the ZOIL Paradigm

To explore OOUI methodologies for the design and implementation of post-WIMP collaborative information interaction, we have developed our own model-based approach. Thereby, we have taken the promising parts from the TASK and OVID methodologies and adapted them to the design of present-day multi-user and multi-surface environments (see chapter 4). Three questions have been guiding our work: Can we adapt OOUI analysis and design techniques and notations to efficiently inform the domain-specific design of present-day interactive spaces? Can we define concise translation rules for creating the initial visual and interaction design for the user interface directly from our model in a simple step-by-step process? How well can designers and programmers apply our OOUI approaches and how do they assess their practical value?

As a testbed for our experimental approach, we have chosen our Zoomable Object-Oriented Information Landscape (ZOIL) paradigm. ZOIL provides a reference interface design for interactive spaces, a reference client-server architecture for distributed information interaction, and a software framework facilitating their implementation. Thus ZOIL provided us with the necessary infrastructure to efficiently explore our model-based approach. The ZOIL reference design, architecture, and framework have been used before in different projects to realize domain-specific prototypes for information interaction. For example Jetter et al. have designed a ZOIL-based user interface for basic personal information management for interactive television devices [12] and two interactive spaces for discussion and presentation, e.g. for students of media science or for
scientists in the field of nano photonics. Heilig et al. have designed an interactive wall for a public library [9]. In future, Geyer et al. will be using ZOIL to create collaborative design rooms for interaction design [8].<fnref n="1" />

A ZOIL-based interactive space consists of several interactive surfaces (e.g. tabletop, tablet PC, wall-sized display) that serve as user terminals to access the shared information space (Fig. 1 top). Each of the terminals thereby provides a window into a much larger planar visual workspace that contains all the shared information and functionality of the application domain. This visual workspace resembles a zoomable whiteboard of infinite size and resolution and is called the “information landscape”. ZOIL’s zoomable information landscape facilitates the navigation in the application domain and its information spaces by “tapping into our natural spatial and geographic ways of thinking” [19]. All domain objects and their relations are organized and visualized in space and scale to foster natural visual-spatial approaches to accessing, sharing, and manipulating information. Regions of the landscape with items, piles, or clusters can represent certain user activities, domain processes, or personal vs. shared information repositories. The landscape is used as a flexible multi-scale medium for visually accessing the application domain and its information spaces and objects. Content and functionality of an individual object can be accessed spatially using panning and “semantic zooming” [19] without the need for opening folders or dedicated applications and the then-necessary management of overlaying or occluding windows (Fig. 2). This zoom navigation is also in line with reality-based interaction: It draws strength from the users’ environment awareness and skills, e.g. their familiarity with approaching, touching, moving, and organizing objects in physical space and the simple fact that “all objects in the real world have spatial relationships between them” [11]. Therefore visual objects at different locations and scales (e.g. virtual Post-It notes, project logos) can further augment the landscape with global or relative landmarks that support orientation. Furthermore, all regions of ZOIL’s landscape can be visually annotated with ink strokes using stylus, touch, or digital Anoto pens on physical paper. Annotations can also be made directly on objects, e.g. slides (Fig. 2).

Multi-user collaboration becomes possible by using ensembles of personal and shared user terminals. All terminals inside the interactive space share the same information landscape. All user-initiated changes to the content of the landscape such as moving, resizing, rotating, or annotating information items are immediately sent to a central server and synchronized with the other terminals in real time (typically within 50-250 ms). However, what region of the landscape is currently visible on each terminal can be individually controlled by the users. For example, users can use a tabletop to interactively zoom into the tiniest details of the landscape at many orders of magnification. At the same time they can display the entire landscape on a peripheral wall-sized screen to provide them with an overview for orientation when needed. The boundaries of the currently visible regions can also be transmitted between terminals. For example, users

<footnote n="1">¹ Videos of these prototypes are available at http://www.vimeo.com/12737554 and http://hci.uni-konstanz.de/jetter/hcse2010.mp4</footnote>
can instruct the remote wall-sized display to zoom and pan to the region of the landscape that is currently visible on the tabletop or vice versa. Thus, by using terminals as “cameras”, the roles of stationary or mobile terminals can be flexibly adjusted by the users depending on the group’s task and preference.

In large information landscapes, users also need efficient ways to find, filter, and analyze single objects or specific clusters. For this reason, ZOIL also integrates physical and virtual “magic lenses” [3] that float above the landscape and through which the underlying content of the landscape can be viewed (Fig. 2). These lenses provide movable filters and visualization tools such as lists, bar charts, scatter plots, or tables to provide an analytical view on the landscape and to facilitate the search and filtering of items using spatial metaphors.

<figure><img src="image_11.png" /><figcaption>**Fig. 2.** *Left:* Semantic zooming into objects in ZOIL uses the available screen estate for smooth changes between iconic representations, metadata display, and full content and functionality, e.g. for viewing, editing, or annotating the content. The example shows a slide object (top) and a movie object (bottom) at different zoom levels. *Right:* Physical or virtual magic lenses allow users to view the underlying landscape using different information visualization tools.</figcaption></figure>

To realize ZOIL’s distributed multi-user and multi-device ZUI, the reference architecture is based on a client-server architecture that provides and synchronizes the data model of the information landscape for all user terminals or clients within an interactive space (Fig. 1). Inspired by Prante et al.’s i-Land with its COAST framework for object distribution [20], we have implemented a dedicated ZOIL server and a client-side data backend as part of our ZOIL software framework for C#/.NET that is based on the db4o object database and its mechanism of *transparent persistence<fnref n="2" />*. For peer-to-peer communication between clients and for input device connectivity, we have chosen the simple but robust stateless *Open Sound Control (OSC)* protocol that can be used for UDP broadcasting within the subnet of an interactive space and enables developers to

<footnote n="2">² http://www.db4o.com/</footnote>
easily integrate novel input devices (e.g. Nintendo Wiimote Controllers or Anoto
digital pens) by connecting to input device middleware such as OpenInterface
[14] or Squidy [13]. Equally important for ZOIL's realization is the framework's
support for fast client-side rendering of complex rich-media zoomable user in-
terfaces. For ZOIL, we have chosen Microsoft's Windows Presentation Founda-
tion (WPF) technology because of following reasons: First, the technology must
support high-performance hardware-accelerated renderings of vector-based user
interface components, so that smooth zooming animations over many orders of
magnification become possible without pixelation. Second, an initial set of fun-
damental user interface widgets such as buttons or sliders, but also more complex
widgets such as video players, document viewers of web browsers should be avail-
able from the start to accelerate implementation. Third, a declarative language
for user interface definition should be available that supports a clear separation
between business logic and visual presentation. In the following we discuss the
central role of WPF's declarative XAML language in our model-based design
and implementation approach.

4 Model-based Design and Implementation with ZOIL

<figure><img src="image_12.png" /><figcaption>Fig. 3. An object-oriented conceptual model of a ZOIL user interface.</figcaption></figure>

For our model-based design and implementation approach, we have employed
an object model similar to the *designer's model* in OVID or the *conceptual user
interface model* in TASK as a core artifact. The model uses a UML-like nota-
tion to define what kind of information objects are visually exposed to the user
and become user manipulatable on the different terminals inside the interactive
space. Furthermore it reveals what attributes or metadata these objects carry
for the user, and what operations or behaviors these objects share and provide.
Fig. 3 shows an example conceptual model for an interactive space in which users can collaboratively explore hotel objects that are contained in ZOIL's zoomable information landscape using semantic zooming. Hotels carry (meta-)data such as the name of the hotel, a photo of the hotel, the country of the hotel, etc. Users can add images from the Web or textual comments as user generated content. Users can explicitly connect all hotels, images, and comments with visual links to structure, annotate, or discuss. It is important to notice that Fig. 3 is not representing the programmer's model of the user interface or its code objects and methods, but that it describes the classes, attributes, and operations of the domain objects and conceptual objects that the user will perceive and act on when interacting with the system's OOUI. "The primary distinction that designers and programmers must keep in mind is that OOUI design concentrates on objects that are perceived by users. OO programming focuses on implementation details that often need to be hidden from the user" [21]. Since the conceptual model is used to inform design and implementation based on human intervention, it is not necessary that the notation completely complies with the UML standard and covers all details. It only has to cover the essentials of the UI from a user's perspective using a notation that has been agreed on and is intelligible for all designers and programmers. In our exploration, we have experienced that our UML-like notation used in Fig. 3 has met these requirements.

We have based our example of a conceptual model on typical user tasks during collaborative holiday planning and an OO analysis of the surrounding information space. Task analysis and OO analysis of the information space revealed the objects and their relations, e.g. whether an object of a certain class should contain or refer to one or many objects of a different class. These relations are specific to the application domain and information space, e.g. the landscape in our example contains 0-n objects of the class *ContentItem*, i.e. hotels, comments, or images. Furthermore, all *ContentItem* objects can be linked to other *ContentItem* object via a *Link* object. The OO analysis of the information space also helps to identify the task-relevant metadata or attributes of an class that should be provided to the users, e.g. alphanumeric fields such as *Name* and *Country* of a hotel, or visual images such as the *Photo* of an hotel. In a ZOIL user interface, objects also carry implicit visual properties such as position, size, and rotation angle that are not provided to the user as numeric values but are used to place and render objects. In Fig. 3 all these different attributes are listed in the middle section of each UML class definition.

After having identified the task-relevant classes, relations and attributes of
objects, the bottom section of the UML class definition is used to specify the
functions or *operations* that objects of this class should expose to the user.
Based on the task analysis, basic operations such as creating, editing or deleting
an object have to be identified and have to be attached to the object itself or
to other user interface objects. For example a virtual Post-It note as *Comment*
object should become editable after zooming in to modify its content. Further-
more users should be able to delete outdated comments. Therefore a *Delete()*
function should be provided to the user that is attached to the object, e.g. a
delete button similar to the close button of a GUI window. However, the func-
tionality to create a new comment *CreateComment(...)* must be attached to the
enclosing visual workspace or screen as the create-functionality must be acces-
sible before the *Comment* object itself exists. Other operations can be mod-
eled for facilitating the zooming navigation, e.g. an object can be assigned a
*ZoomItemToFullScreen()* functionality to offer an automated zooming that re-
veals all attributes, metadata, and operations by a simple tap or click on the
object. While modeling the operations of objects, further design decisions have
to be made, e.g. whether an object is movable, resizable, or rotatable. Also the
functionality that should be executed when using drag-and-drop manipulations
can be modeled. For example, the behaviors section of a class can define what
should happen as soon as an object of a certain class has been dropped onto it,
e.g. creating a link to the dropped object in *CreateLinkToDroppedObj(...)*.

The design of the conceptual model should be accompanied by two continuous activities to ensure its quality: First, choosing appropriate class hierarchies is essential for the OOUI's coherence and consistency. Therefore the model should be continuously checked if all new commonalities in attributes and operations have been modeled in common base classes. Second, during OOUI design the higher level task models have to be decomposed into sequences of lower level direct manipulations of objects and other invocations of their operations. In many cases it is not immediately visible if a model covers all required tasks and therefore this should be frequently verified. This can be achieved by manually simulating a user task and using the conceptual model for a sequential walk-through that checks if all necessary objects, attributes, and operations for all tasks are available.

## 4.1 Model-based Design and Implementation of UI Objects

ZOIL's reference design and architecture provide a generic design and implemen-
tation framework in which only the application-specific details of the user inter-
face and interaction design have to be fleshed out. Our model-based approach
provides the necessary translation rules in a simple step-by-step process, thereby
allowing designers to create initial sketches of visual and interaction design from
the conceptual model of the user interface. It furthermore enables designers and
programmers to easily turn the resulting sketches into an implementation model
for the user interface object based on XAML. This XAML code can then be
used to test the design as an interactive prototype. Our model-based translation
process can be described as a four phase process and is visually illustrated in
Fig. 4 for the example of a *Hotel* object.

The first step of the translation process is to decide which attributes and
operations of an object should appear on which level of semantic zooming. At-
tributes or operation can either appear globally at all zoom levels or they can be
assigned to different zoom levels, so that they only appear or become active after
the user has zoomed in. In Fig. 4, the *Delete()* function is global and appears at
all levels of detail. This is also true for the manipulation of the object's position,
rotation angle, or size (*Move()*, *Rotate()*, *Resize()*) and its functionality to react
to objects that have been dropped onto it such as `CreateLinkToDroppedObj()`. The most important attributes that a user frequently needs to recognize or recall an object (e.g. `Photo` or `Name` of an hotel) already appear at small zoom levels in the early stages of zooming. The attributes only necessary for more in-depth exploration (e.g. `Stars`, `Country`, `Price`) appear after enough screen estate is available, e.g. on zoom level 3. This is also true for advanced functions like `ShowOnWallScreen()` that shows a hotel on a shared wall-sized display.

In the second step, this assignment is used to sketch the global appearance and behavior of the object (Fig. 4 top right). The different operations and their triggering manipulations or widgets are modeled using simple sketches: In our example, the typical multi-touch gestures known from tabletops or smart phones are used for `Move()`, `Rotate()`, and `Resize()`. A zoom-to-full-screen animation is issued by a single tap with the finger on an object (`ZoomItemToFullScreen()`). Another item can be dragged on the object with the finger, activating the `CreateLinkToDroppedObj()` functionality if the item is of the type `ContentItem`.

In the third step, the individual zoom levels are sketched based on the assignments of attributes and operations from step one (Fig. 4 right). These sketch models are created for each zoom level to move from conceptual design to the concrete design of the visual appearance of objects. Since the necessary attributes and operations for each zoom level are known, the complexity of the design task is minimized and can be carried out with standard techniques.

In the final step, the sketch models of the different zoom levels are translated into the implementation model of the user interface object (Fig. 4 bottom). This translation is supported by ZOIL's software framework that extends the declarative XAML user interface description language of WPF with ZOIL-specific elements. By introducing ZOIL's *ZComponent* user interface control, an object's appearance at different semantic zoom levels can be defined entirely using declarative approaches (similar to HTML) without the need for procedural programming. The different zoom levels are managed by ZOIL's *ZComponent-Frames* container that selects the appearance of an object depending on the available render size. To avoid harsh visual changes, zoom levels smoothly blend between two appearances using an opacity animation. Furthermore designers and programmers can easily assign predefined ZOIL behaviors to an object using the *attached behavior* software pattern<fnref n="3" />. This pattern helps to encapsulate frequently used ZOIL-specific behaviors (e.g. "object can be manipulated with multi-touch", "object zooms to full-screen after tap", "object is a target for dropping another object") in a central behavior library. Behaviors from the library can be easily attached to classes or individual instances of objects using declarative XAML code without the need to know procedural programming or to fully understand the underlying class hierarchies. We believe that this combination of the *ZComponent* object and the attached behavior pattern introduces a great expressive power to the declarative XAML language and a very natural view of interactive behavior into user interface programming. It greatly facilitates the

<footnote n="3">³ http://blogs.msdn.com/b/johngossman/archive/2008/05/07/the-attached-behavior-pattern.aspx</footnote>
translation of sketch models with their visual appearance and behavioral proper-
ties into implementation models. As illustrated in the implementation model in
Fig. 4, the process of translating a sketch model in XAML is thereby a straight-
forward task that does not rely on advanced programming skills.

## 4.2 Case Study

In order to investigate the utility and applicability of our OOUI approach in
practice, we conducted a case study with 11 participants (9 graduate-level and
2 undergraduate students of computer science). The question guiding our study
was how well participants can apply our approach and how they assess its prac-
tical value during a small-scale project. We divided the participants into five
teams (4 teams with 2 members, 1 team with 3 members). In a first one-hour
session we presented our modeling approach to all teams: We created and ex-
plained a conceptual model of a ZOIL user interface for accessing a fictitious
image database. The teams were then given the assignment to create an own
conceptual model for a different ZOIL user interface until the next session in
two weeks. The user interface to model should allow users to explore and discuss
hotels as described in the example in the previous sections. We provided the
teams with the same input for their modeling and design activity that we used
ourselves to create the example model in Fig. 3, i.e. all teams were handed 8
informal functional requirements (e.g. "user must be able to add a textual com-
ment to the workspace") and a list of 22 required object properties (e.g. "each
*Comment* has an *Author*", "each *Image* carries *Tags*").

Two weeks later, we carried out individual one-hour team sessions during
which each team completed three tasks. First, each team presented and explained
their prepared conceptual model. Then we asked the team to check if their model
really supports the 8 functional requirements by carrying out a walkthrough.
We then presented the team our alternative model (Fig. 3) and asked them to
validate this unknown model by another walkthrough. After this, each team
member filled out a questionnaire to rate the difficulty of the three tasks. At
the end of the sessions, the teams were instructed to design and implement a
user interface with the ZOIL framework based on Fig. 3 until the next sessions
in the following week. In these last sessions, each team individually presented
the resulting interactive prototype and each team member filled out a further
questionnaire to rate the overall usefulness of the modeling approach and the
difficulty to apply it on user interface design and implementation.

During the case study, all teams presented conceptual models that were for-
mally correct and supported the 8 functional requirements. All teams were able
to carry out a walkthrough to validate their own and unknown models. Fur-
thermore, the presented interactive prototypes covered the requested function-
ality. However, during the first and second session participants reported initial
problems regarding the unfamiliar use of UML class diagrams to model user
interfaces. Repeatedly participants mentioned that they sometimes had fallen
back into the familiar modeling of code objects and lost track of their original
intention to model the user interface from a user's perspective. However, the
<figure><img src="image_13.png" /><figcaption>**Fig. 4.** ZOIL's translation process and rules to translate the object model to user interface design and implementation.</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Fig. 5. Collected feedback from the questionnaires of the case study.</figcaption></figure>

participants reported that they got increasingly used to the approach and found it useful to support the design and implementation. Fig. 5 shows the results from the questionnaires: the creation of a model (mean=3.45, sd=0.93) and checking the own or someone else's model with a walkthrough (mean=3.1, sd=2.9 and mean=2.9, sd=1.14) was not considered as particularly difficult nor very easy. This is rather encouraging, as the students were given only a very brief introduction to the approach without any proper training phase. Furthermore, the overall utility of the modeling technique was considered as useful (mean=4.1, sd=0.99) by the participants. Regarding the early stage of our approach and the unfamiliar use of object-oriented modeling and design for user interfaces, we consider these results as a promising first evidence that OOUI approaches can be indeed useful for designing interactive spaces.

## 5 Conclusion and Future Work

We have discussed why we believe that revisiting OOUIs has a great potential for the design of future post-WIMP environments, particularly for collaborative information interaction. We have introduced our ZOIL paradigm that we have used as an experimental testbed for creating and evaluating our OOUI approach. We have illustrated and discussed our approach in detail and have shown how it can efficiently inform the design and implementation of user interface objects following simple translation rules. Furthermore, we have presented promising results from a first case study on the practical utility of our approach. At the current stage, we consider our approach as a successful first step. However, ZOIL-based interactive spaces offer a great design space and currently only small parts of it have been covered by our approach. For example, the design of ZOIL's magic lenses, visualization tools, or the integration of physical objects or paper is not covered yet. Therefore we will investigate how new and extended modeling notations and translation rules can be used to cover these aspects in future.

## References

1. Common User Access Guide to User Interface Design. IBM Corporation (1991)
2. Beck, A., Janssen, C., Weisbecker, A., Ziegler, J.: Integrating object-oriented analysis and graphical user interface design. In: Software Engineering and Human-Computer Interaction. vol. 896, pp. 127–140. Springer Berlin / Heidelberg (1995)

3. Bier, E.A., Stone, M.C., Pier, K., et al.: Toolglass and magic lenses: the see-through interface. In: Proc. SIGGRAPH '93. pp. 73–80. ACM, New York, NY, USA (1993)

4. Blandford, A., Attfield, S.: Interacting with information. In: Carroll, J.M. (ed.) Synthesis Lectures on Human-Centered Informatics. Morgan & Claypool (2010)

5. Collins, D.: Designing object-oriented user interfaces. Benjamin Cummings, Redwood City, CA (1995)

6. Constantine, L.L., Lockwood, L.A.D.: Software for use. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA (1999)

7. Fitzmaurice, G.W., Khan, A., Buxton, W., Kurtenbach, G., Balakrishnan, R.: Sentient data access via a diverse society of devices. Queue 1(8), 52–62 (2003)

8. Geyer, F., Reiterer, H.: A cross-device spatial workspace supporting artifact-mediated collaboration in interaction design. In: Proc. CHI EA '10. pp. 3787–3792. ACM, New York, NY, USA (2010)

9. Heilig, M., Mischa, D., Rexhausen, S., Huber, S., Runge, O.: Search, explore and navigate - designing a next generation knowledge media workbench. In: Proc. SIDeR '09. pp. 40–43. Eindhoven University of Technology, Eindhoven, NL (2009)

10. Hutchins, E.L., Hollan, J.D., Norman, D.A.: Direct manipulation interfaces. Hum.-Comput. Interact. 1(4), 311–338 (1985)

11. Jacob, R.J., Girouard, A., Hirshfield, L.M., Horn, M.S., Shaer, O., Solovey, E.T., Zigelbaum, J.: Reality-based interaction: a framework for post-wimp interfaces. In: Proc. CHI '08. pp. 201–210. ACM, New York, NY, USA (2008)

12. Jetter, H.C., Engl, A., Schubert, S., Reiterer, H.: Zooming not zapping: Demonstrating the zoil user interface paradigm for itv applications. In: Adjunct Proceedings of EuroITV 2008. Springer (2008)

13. König, W.A., Rädle, R., Reiterer, H.: Interactive design of multimodal user interfaces. Journal on Multimodal User Interfaces 3(3), 197–213 (Feb 2010)

14. Lawson, J.Y.L., Al-Akkad, A.A., Vanderdonckt, J., Macq, B.: An open source workbench for prototyping multimodal interactions based on off-the-shelf heterogeneous components. In: Proc. EICS '09. ACM, New York, NY, USA (2009)

15. Lehikoinen, J., Aaltonen, A., Huuskonen, P., Salminen, I.: Personal Content Experience: Managing Digital Life in the Mobile Age. Wiley, Chichester, UK (2007)

16. Mandel, T.: The GUI-OOUI War, Windows vs. OS/2: the designer's guide to human-computer interfaces. Van Nostrand Reinhold, New York, NY, USA (1994)

17. Memmel, T., Reiterer, H.: Model-based and prototyping-driven user interface specification to support collaboration and creativity. J.UCS 14(19), 3217–3235 (2009)

18. Pawson, R., Matthews, R.: Naked objects: a technique for designing more expressive systems. SIGPLAN Not. 36(12), 61–67 (2001)

19. Perlin, K., Fox, D.: Pad: an alternative approach to the computer interface. In: Proc. SIGGRAPH '93. pp. 57–64. ACM, New York, NY, USA (1993)

20. Prante, T., Streitz, N., Tandler, P.: Roomware: Computers disappear and interaction evolves. Computer 37(12), 47–54 (2004)

21. Roberts, D., Berry, D., Isensee, S., Mullaly, J.: Designing for the User with OVID. Macmillan Technical Publishing (1998)

22. Shaer, O., Jacob, R.J.: A specification paradigm for the design and implementation of tangible user interfaces. ACM Trans. Comput.-Hum. Interact. 16(4), 1–39 (2009)

23. Wigdor, D., Jiang, H., Forlines, C., et al.: Wespace: the design development and deployment of a walk-up and share multi-surface visual collaboration system. In: Proc. CHI '09. pp. 1237–1246. ACM, New York, NY, USA (2009)
# ViSE – A Virtual Smart Environment for Usability Evaluation

Stefan Propp, Peter Forbrig

University of Rostock, Institute of Computer Science,
Albert Einstein Str. 21, 18059 Rostock, Germany
{stefan.propp, peter.forbrig} @ uni-rostock.de

**Abstract.** Within the research field of HCI task models are widely used for model-based development of interactive systems. Recently introduced approaches applied task models further to model cooperative behaviour of people interacting in smart environments. However there is a lack of usability methods to support the needs of evaluations during a model-based development process for smart environments. Particularly during early stages of development building a prototypical environment for user evaluations is resource consuming. To overcome the challenges we present a process model and according tool support. We provide the virtual smart environment ViSE to conduct expert evaluations and user studies during a user-centred design process, supporting iterative evaluations.

**Keywords.** Model-based Usability Evaluation, Task Models, Smart Environment

## 1 Introduction

According to Weiser’s vision [9] of ubiquitous computing, devices are weaving themselves into everyday life, allowing people to fully concentrate on performing their tasks, while hiding complexity of necessary devices. A smart environment (SE) recognizes user behavior and provides assistance to achieve the users’ objectives. For instance within a meeting scenario the presenter should concentrate on the talk, while the SE assists by adjusting the projector and capturing audiovisual data for meeting documentation if needed.

While offering a higher degree of comfort, also new challenges are introduced. In contrast to desktop computing, where a user focuses the attention on a single device, in a SE multiple devices influence user behavior. Reflected from a task-oriented point of view, a certain task can be started at one device, while being finished at another device. For instance during a discussion some comments are quickly typed into a PDA, being revised later back in the office at a notebook. Different user interactions lead to a wide variety of options [7] for the design of the SE, comprising modalities (like speech or gestures), device selection (like direct access by a user or dynamic
selection by the SE) and the initiative (explicit or implicit). An example for explicit initiative is a user manually turning the light on via switch, whereas an example for implicit initiative describes a scenario where the SE senses that it is too dark and decides autonomously to turn the light on. As we can see, an interaction needs to be interpreted within a certain context. Contextual conditions like available devices, currently present users, light or temperature, may influence task performance for each user. Furthermore cooperation is an important aspect. Within SEs a certain task may be performed cooperatively by several users, who reflect certain roles.

These characteristics of SEs impose challenges for usability evaluation. An evaluation is particularly resource consuming. To conduct a user study within a real SE a completely functional prototype has to be set up, while evaluating a simulated environment needs effort for instance to prepare a "Wizard of Oz"-experiment.

For model-based development of SEs some approaches apply task models [1, 2, 3, 8, 10]. We suggest reusing these task models also for usability evaluation to integrate evaluation into development. Advantages are (a) the reduced effort to prepare evaluations rapidly at all stages of development, (b) an easier interpretation of task-based log files than of low level events and (c) the direct link back to the development models facilitates their improvement.

## 2 Related Work

Several recently published approaches apply task models for model-based development of SEs. Trapp et al. [8] define each device's capabilities with a task model chunk. When a new device connects to the room infrastructure, the corresponding model chunk is added to the room task model. As a result combined functionalities may be offered. For instance a scanner and a printer may offer an additional copying functionality. Wurdel et al. [10] model behaviour of persons with task models and describe their collaboration. For instance, that person "A" has to finish the talk first to give person "B" the floor. Each task is described by a set of preconditions and effects on the environment. Feuerstack et al. [1] enhance the task modelling notation CTT to serve as model for runtime interpretation. Domain concepts are annotated and the object flow is modeled. Different users' task models are synchronized with domain objects. Luyten et al. [3] focus on modelling distributed user interfaces for dynamic environments. Task and environment models are visualized to reduce complexity.

While several approaches for the development of SEs are available, there are only a few appropriate usability methods. Scholtz et al. [6] suggest a general framework with evaluation metrics to give a starting point for structuring the evolving evaluation techniques. Maly et al. [4] describe an approach of visualizing the user's behaviour as 3D representation and allow experts to create artificial scenarios when sensor data is not available yet. However, task models are not used. In [5] we suggest a task model-based process for evaluating smart environments. Performed tasks are visualized within animated task models and serve for analysis. In this paper we extend the approach to build a virtual smart environment, which visualizes user interactions as a 2D representation of a physical SE.
# 3 Usability Evaluation Method

Software development is structured into several stages, comprising requirements analysis, design and implementation of the final product. Accordingly we describe a structured process of development and evaluation of SEs. This section presents the process model (section 3.1) and subsequently discusses the usage of ViSE to conduct expert evaluations (section 3.2) and user studies (section 3.3).

## 3.1 Process Model

The process model (fig.1) describes an approach for the iterative development and evaluation of SEs. During each development cycle three stages are considered.

1. During planning stage the development methods and evaluation techniques are chosen. If any developed artifacts of a previous cycle exist, their evaluation results are taken into account.

2. During development stage existing artifacts are refined and new artifacts are developed. Examples are requirements documents, models and source code.

3. Iterative evaluations comprise direct feedback for the developed artifacts based on feedback of user studies and expert evaluations.

<figure><img src="image_15.png" /><figcaption>Fig. 1. Process model for design and evaluation of Smart Environments</figcaption></figure>

Depending on progress within development different activities are carried out during each cycle, which should be subsequently outlined.

**Requirements analysis.** To elicit requirements a dialog with customers and prospective users is conducted. We identify user roles and create a task model for each role. A task model captures task hierarchies and their temporal relations. To provide additional insights (e.g. context dependencies) textual scenarios are captured, which may serve as test cases during evaluations.

During evaluation the consistency of the designed task models has to be analyzed. A task model animation tool allows interactively walking through the models.
Another challenge is to evaluate the users’ stated assistance functionalities, which the
smart environment should provide, like automatically switching lights on or showing
presentation slides of a presenters’ laptop. We provide a virtual smart environment
(ViSE), which allows conducting “Wizard of Oz”-Experiments for SEs: An expert
hides in another room, observing user interactions via video camera. The expert
operates the devices within the physical environment remotely to mimic the
envisioned assistance. Changing devices’ states in ViSE allows changing the devices’
states in the physical SE accordingly. Involved users get a feeling of the envisioned
assistance and reflect their requirements

**Design.** Within the design stage task models have to be enriched with further details to more accurately reflect user behavior and to serve as initial models for implementation. Textual scenarios elicited during requirements stage contain additional details about cooperative dependencies between different users, like a chairman is required to give a presenter the floor. Further details comprise contextual dependencies for performing tasks, like having a laptop containing slides to give a presentation. These aspects are modeled in CTML ("collaborative task modeling language") [10]. Each task is annotated with preconditions and effects. A precondition formalizes the environments' state before a particular task can be performed. An effect describes the environments' state after a task is performed.

To ensure the consistency of the designed models, the virtual SE visualizes the modeled SE and allows an interactive walk-through as collaborative animation. Scenarios from requirements stage are animated by a usability expert and identified inconsistencies can be corrected.

**Implementation.** Based on the designed task models the SE's software has to be implemented. The envisioned SE assists users while performing their tasks and proceeds in three steps: firstly it senses user movements and further context information (e.g. location sensors to separate presenter and audience), secondly it infers the task currently performed (e.g. a person is beginning to present) and finally it triggers devices accordingly (e.g. a projector performs a system task "show agenda"). Within our physical SE this recognition of performed tasks is accomplished by a probabilistic behavior model, which is derived from previously specified task models as discussed in [2]. Finally the software is deployed at the devices within the environment.

To ensure an adequate integration of all software and hardware components, user
studies have to be conducted. During evaluation a vast amount of sensor data within
the SE is captured, leading to the challenge of a suitable interpretation of user
behavior to discover usability issues. Our virtual SE allows playing captured data
forth and back to interactively explore issues.

After having introduced the process model and provided tool support we take a closer
look at expert evaluations during design stage and user evaluations during
implementation stage.
## 3.2 Expert Evaluation of cooperative task models

After having specified a CTML design model its consistency according to the elicited requirements has to be evaluated. Fig. 2 depicts ViSE within an expert evaluation.

The left part of ViSE contains a birds' view of the environment, comprising users, devices (e.g. laptops and projectors) and other objects (e.g. pens and whiteboard erasers). Interactions with these entities allow changing the SEs' state and performing tasks. The following interactions are supported:

* Changing the persons' location
  (e.g. moving to an empty seat)

* Changing the persons' corpus orientation
  (e.g. to look to the front, where the presenter is speaking)

* Attaching items to persons
  (e.g. a presenter is taking up a laptop)

* Changing devices' states
  (e.g. switching a laptop on)

* Establishing connections between devices
  (e.g. connecting a laptop to a VGA port to connect it to the room infrastructure)

The upper right part contains the persons within the current scenario with their associated roles. For each role a task model is animated. A task can be performed if both the temporal relations and the preconditions are fulfilled.

The lower right part contains the preconditions for the current task and indicates which conditions are currently “true” and “false”. For the task “present with projector” three preconditions are specified (see fig. 2, lower right part):

* The person with the role “presenter” has to be located at the “PresentationZone”.

* The person has to carry a device of type laptop or PDA.

* The persons' device has to be connected to a VGA port.

In the scenario in fig. 2 requirements engineer Stefan fulfills all three preconditions and the temporal relation that he has already performed task “enters room”. Hence he is allowed to perform the task “presents with projector”.

If any preconditions are differing from the requirements, they can be edited textually to directly improve the evaluated artifacts. As an alternative preconditions can be edited graphically. A designer may move persons to a certain location and equip them with devices. A subsequent option “generate preconditions” allows updating the preconditions accordingly.
6

<figure><img src="image_16.png" /><figcaption>Fig. 2. Interactive walk-through of a design model with ViSE</figcaption></figure>
## 3.3 User Evaluation of a physical Smart Environment

After all components are set up, a user study can be conducted to evaluate the physical SE. Users are invited to perform tasks specified in a test plan as individual tasks or within a cooperative scenario. At the same time user interactions are captured: user movements via UbiSense location sensors, handled devices or other items via RFID tags, performed tasks via task recognition algorithms and video streams via cameras. Most data is already captured for normal operation of the SE. Hence it is available for evaluation.

<figure><img src="image_17.png" /><figcaption>Fig. 3. Replay of sensor data with ViSE</figcaption></figure>

ViSE was enhanced with a replay mode (fig. 3), which allows replaying captured sensor data with different animation speeds to provide insights into user interactions. A birds' view visualizes user movements, depicting locations of users and devices as according icons and way points reflecting past location sensor data with a particular focus on locations where tasks were performed. Graphically selecting users, devices or way points of user movements gives further information as tool tip (e.g. which task was performed at a certain way point) and filters sensor data. For instance spanning a selection rectangle over the presentation zone in front of the audience delivers an overview of all performed tasks within that area. During replay further expert observations can be annotated. For instance currently our physical SE recognizes only a limited set of tasks automatically. Additional tasks can be identified manually during video replay. According replay controls are depicted within a view at the right of fig. 3. Further views provide several logs and visualizations of sensor data on demand, for instance the progress of task performance as animated task model or gantt chart.
## 5 Conclusion

In this paper we have given an overview of model-based development of SEs and shown how to exploit designed task models for usability evaluation. A process model describes a process of iterative development and evaluation. The virtual smart environment ViSE provides tool support at requirements, design and implementation stage. In early development stages, when a physical SE for user studies is not available, ViSE allows conducting expert evaluations for SEs. After a physical SE is set up ViSE helps to cope with the vast amount of captured sensor data. We emphasize on providing an integrated view on task performance, a graphical presentation of user and device movements and video streams, for both presenting and interactively exploring usability data.

Future research avenues comprise gathering further real life experiences to improve the presented method and tools.

**Acknowledgments.** The work of the first author was supported by a grant of the German National Research Foundation (DFG), Graduate School 1424.

## References

1. Feuerstack, S., Blumendorf, M., Albayrak, S.: Prototyping of Multimodal Interactions for Smart Environments based on Task Models. AMI'07 Workshop on Model-Driven Software Engineering, Darmstadt, Germany, 2007.

2. Giersich, M., Forbrig, P., Fuchs, G., Kirste, T., Reichart, D., Schuhmann, H.: Towards an Integrated Approach for Task Modeling and Human Behavior Recognition. HCII 2007, Beijing, China, 2007.

3. Luyten, K., Vandervelpen, C., Coninx, K.: Task modeling for ambient intelligent environments - design support for situated task. Tamodia 2005, Gdansk, Poland, 2005.

4. Maly, I., Curin, J., Kleindienst, J., Slavik, P.: Creation and Visualization of User Behavior in Ambient Intelligent Environment. Information Visualisation, London, Great Britain, 2008.

5. Propp, S., Buchholz, G., Forbrig, P.: Task Model-based Usability Evaluation for Smart Environments TAMODIA 2008, Pisa, Italy, 2008.

6. Scholtz, J., Consolvo, S.: Towards a Discipline for Evaluating Ubiquitous Computing Applications, Technical Report of Intel Research, 2004.

7. Shirehjini, A.: A Multidimensional Classification Model for the Interaction in Reactive Media Rooms. HCII 2007, Beijing, China, 2007.

8. Trapp, M., Schmettow, M.: Consistency in use through Model based User Interface Development. Workshop at CHI'2006, Montreal, Canada, 2006.

9. Weiser, M.: The Computer for the 21st Century. Scientific American, 265, pp.94-104, 1991.

10. Wurdel, M., Propp, S., Forbrig, P.: HCI-Task Models and Smart Environments. HCIS 2008, Mailand, Italy, 2008.
# A Domain Specific Language for Contextual Design

Balbir S. Barn and Tony Clark¹

Middlesex University, Hendon, London, UK, NW4 4BT

**Abstract.** This paper examines the role of user-centered design (UCD) approaches to design and implementation of a mobile social software application to support student social workers in their work place. The experience of using a variant of UCD is outlined. The principles and expected norms of UCD raised a number of key lessons. It is proposed that these problems and lessons are a result of the inadequacy of precision of modeling the outcomes of UCD, which prevents model driven approaches to method integration between UCD approaches. Given this, it is proposed that the Contextual Design method is a good candidate for enhancing with model driven principles. A subset of the Work model focussing on Cultural and Flow models are described using a domain specific language and supporting tool built using the MetaEdit+ platform.

## 1 Introduction

This paper examines the role of user-centered design (UCD) to the design and implementation of a mobile social software application for supporting student social workers in their work place. The principles and expected norms of UCD raise a number of issues which lead us to propose that these problems are a result of the inadequacy of precision of modeling the outcomes of UCD, which prevents model driven approaches to method integration between UCD and established software engineering practice. A particular UCD approach - Contextual Design [3] is explored in detail from a model/language design perspective by first critiquing the key issues of ambiguity and lack of precision of diagrams normally produced as a result of Contextual Design activities. Following on from this, a subset of Contextual Design, namely, the Cultural Model is developed in terms of abstract and concrete syntax together with its accompanying semantics diagram using an approach to language design described by Clark et al [7]. An implementation using the MetaEdit+ tool [17] is also briefly described. The issues of a lack of precision of UCD methods represents an ongoing research challenge in the field of requirements engineering and a key outcome of this paper is to encourage discussion of these problems and lessons to enable method re-engineering of UCD practice.

The paper contributes to current research in human centred software engineering by providing a formal syntax and semantics for aspects of the Contextual Design methodology and in doing so provides a route whereby the exploration of how UCD and software engineering can be integrated.
The remainder of the paper is structured as follows: section 2 introduces key aspects of UCD and model driven development; section 3 presents the key motivation of this work, our experience of using UCD and the lessons learnt, and provides a short introduction to Contextual Design, the UCD approach we have selected to be subject to a formal treatment; section 4 puts the case for using a *model driven* approach to UCD; section 5 presents the approach we have taken to develop a modelling language with more precise syntax and semantics; section 6 presents the domain specific language version for Contextual Design along with an illustrative example of its use; section 7 presents concluding remarks and notes for further research.

# 2 Background to UCD and MDD

## 2.1 From User-Centered Design to Participatory Design

A detailed review of the literature concerning user-centered design is not possible within the constraints of this paper but it is useful to present an overview of key phases in development of user engagement in systems design processes. User-Centered Design (UCD) or the variant, User-Centered Systems Design [22] emerged in the 1980s as an important development recognizing the move from batch computing to interactive computing applications where there was a need to involve users in the design process. At that time, however, as Marti and Bannon [20] indicate: UCD did not imply that “users were ... active participants in the design process”, rather they were studied, observed, measured as a way of gathering requirements for the system development [8]. An implication of UCD is thus one of where the designer (hopefully) reacts to feedback from the user. A more radical school of user involvement is that attributed to the so-called Scandinavian Model of UCD, namely, Participatory Design (PD) that emerged from the research activity of people such as Bjerknes et al [5]. In PD, users are seen as equal partners in the design and development of systems. This involvement of users implies users as “active agents” and later became known as Cooperative Design [11] or more latterly as “Co-Design”. In PD, interestingly, there is a focus on primary work processes and identification of technology to enhance and better support work activities. (The basis of business process modeling). As UCD concepts became established they were further elaborated as ISO Standard 13407, Human-centred design processes for interactive systems [23]. These concepts were developed and extended into 12 key principles for UCD by Gulliksen [15].

While the mantra of involving users in the design process is now well ingrained [26] it has been contested and more recently Marti and Bannon [20] outline caveats where they argue that involving users can present problems. The characterization of problems they have identified forms part of the evaluation of our experience of co-design when applied to our development of a mobile application for e-learning, and which led us to consider how such issues may be addressed by model driven practice for UCD.
## 2.2 Model Driven Development

Orthogonal but related to UCD is the need to recognize that software engineering development methods have also evolved and more recently model driven development is increasingly seen as critical to good design: see [10] for an overview of MDD where it is argued that modelling is a key technology that is necessary to address the representation gap between human understanding of complex modern systems and their implementations and where precision at all levels of development is key to the increased scope of computer-based support for systems development. We argue that precision is key to increasing all aspects of system quality including reliability, usability, efficiency, and that MDD offers an approach that provides precision from a range of appropriate perspectives. MDD is increasingly being used for user-centred aspects of systems such as HCI [27,25,24] and safety [1].

Modeling in general is viewed as a capstone of many software engineering approaches where it is used to as an approach to user requirements definition and as a basis for developing information systems to meet those requirements. Models provide a vehicle for explaining and sharing understanding of complex problems and provide capabilities for different views of the underlying problem at different levels of abstraction. Model driven architecture takes this premise further by providing an overarching conceptual structure for using and applying transformations to models in a structured and controlled manner in all stages of the software engineering development process.

The Object Management Group (OMG) provides a set of standards to express models and model-model transformation and has been leading industry initiatives in the promotion of technologies, methods and standards under the banner of model driven architecture (MDA) [13]. Our position is: MDA has key role to play in systems development and are in agreement with Constantine and Lockwood [8], who assert that UCD can be ambiguous and vague. In contrast, Gulliksen et al assert that “model driven approaches represent a move away from user-centered design reducing their involvement to that of the users being informants rather than co-designers”. This assertion needs re-visiting in the light of MDA approaches to user interface design and recent advances in domain specific languages. Certainly Fisher [9] has identified that Collaborative Design and meta-design (using MDA principles) are key themes facing software engineering research and practice.

## 3 Experience of User Centered Design

This section describes some of our experiences from a recent research project that utilised UCD and software engineering approaches to developing a mobile application for Social Work education. We discuss some of the key issues and lessons arising from that experience and present an argument for model driven UCD.
**3.1 Case Study**

The motivation for exploring how model driven principles could be applied to user centred design arose from a recent research project where we applied a variant of user centred design to design and implement a mobile device application to support Social Work Education in the UK.

In common with many other professions, the training of social workers requires students to be placed in social work settings and to undergo assessment in the workplace. Trainee social workers in England (those on an accredited social work degree (UG or PG) must successfully complete 200 days in a practice setting. Such placement can occur in different size blocks according to structures and requirements of individual degree programmes. These requirements are maintained and regulated by the Social Work professional body – the General Social Care Council (http://www.gscc.org.uk/Home/).

During the practice learning process, there are several key stakeholders involved, including: the student; the practice mentor and assessor; the University academic tutor; the work based supervisor. The key outcome of the placement is a report that outlines the skills and competencies raised along with supporting evidence collected from the placement.

Given these background concepts the research project aimed to develop a set of applications both mobile and web-based that supported student social workers in the planning and design of practice learning assessments and in the collation of research and practice evidence towards a final report.

**3.2 Experience using UCD**

The project team assigned to the project was multidisciplinary. There were academic experts from Computer Science, Sociology, Social Work, along with practitioners from the Social Work field. Further, the project development teams were located in multiple locations across the UK South. As well as the multiple disciplines located within the team, the Computer Science team further represented alternative approaches to systems design, with representation from both MDA and UCD. These different approaches led to some creative tension manifested in early debates similar to that discussed between Gulliksen and Constantine. Given the make-up of the project team, it was essential to agree to a methodology that could accommodate disparate views. The team had previous experience of using a co-design process for developing mobile applications for the Nursing domain [21]. Hence this approach was adapted to suit the needs of this project and the software engineering principles influencing members of the team. Thus the project deployed a variety of methodological techniques that draw upon software engineering, social sciences research and usability.

**3.3 Problems Encountered**

While the system was successfully developed, its deployment and use was very limited, and is consequently still ongoing (past the project completion date).
This is partially attributed to the implementation of the co-design approach and
it is here that it is considered that there are many lessons to be learnt. Using
the putative framework of problems identified by Marti and Bannon [20] as a
starting point the following lessons are presented:

**user types** The intended software applications were designed for several types of users.

**users as designers** While it is accepted that all users can design at some level – that is have ideas, think creatively about different uses of tools and convey those thoughts in some form explicit knowledge transfer – it is clearly not the case that users have the necessary design skills to engage in all stages of the design process.

**new technologies** Mobile technology is evolving at a rapid pace. Increasing power, capability and software applications possible makes it very difficult for non-technologists to remain abreast with such change. In order for users to make a significant contribution to the design process they need to have a logical understanding of technological solutions in order to be able to conceptualize new scenarios of use. This problem manifested itself very early in the co-design process: many of our participants had their first direct contact with current mobile devices in our show and tell workshops.

**work environments** Project champions tend to be located at management level where there is often limited understanding of operational requirements. This can have a detrimental effect on active user involvement throughout the design process. The Social Work environment in a public sector setting meant there were work pressures that often prevented users from securing sufficient time to effect a meaningful engagement in the co-design activities.

**deployment risk** The need for sufficient training, guidance, support and impact assessment in the work environment also need to be sufficiently defined. Issues of risk and technology in the workplace, although correctly identified at the beginning of the project manifested themselves resulting low usages in the work place context. There has been considerable interest within the social sciences in developing ideas related to risk. Beck in his seminal text Risk Society [2] argued that the “technisisation” of risk derives from the omnipresence of technology. The experience here could be seen as reflecting fear on a number of dimensions. Reluctance to engage with the project could be seen as a fear of technology itself and the ability of some individuals to cope with technological demands. The social work task is in itself high risk and high profile and the use of technological devices for training purposes could be seen as representing a ‘reflexive’ form of risk. The findings could also reflect the so-called ‘precautionary principle’. Practitioners and students were anticipating possible difficulties in areas such as confidentiality and data protection which prevented them from considering the possible full benefit of the opportunity offered by the project.

**user confusion** Confusing what users want with what they truly need. Numerous user studies and approaches can create a wide and detailed user understanding however such studies can create confusion with what users say they want with what they really need.
**multi-faceted design team** The make-up of the design team can influence the nature of user involvement. For example, a team that is equipped with skills in UI, prototyping, and software design will likely involve users at stages in the design process. A team with predominantly HCI researchers will likely involve more users and at more stages. In this case, there was a relatively balanced team in terms of skills and knowledge – our problems were arriving at shared common vocabularies, and attempting to involve all users and all design team members all the time.

# 4 The Case for Modelling in User Centered Design

These lessons or observations from the co-design approach have the potential to be mitigated by taking a model-driven integrated approach to the artifact development from the co-design activities. This paper argues that artifacts from user-centred design should be model based so that transformations between viewpoints can be integrated. This requires a user-centered design approach that is both rich, for capturing key user requirements and is also model driven such that it can be subject to model driven transformations during the design and implementation process.

Currently, UCD approaches are strong on user engagement and communication but tend not to be model-based in the software engineering sense. Thus it is difficult to derive a single viewpoint to meet both the needs of stakeholders and software engineers. Such design-slicing could be a powerful feature in presenting key features of an overall design without information overload.

Multiple viewpoints are a recognized approach to such a challenge but tend to be driven by software engineering needs. For example, Rational Unified Process [18] has attempted to integrate user centered design activities. Such models serve software engineering well but present notational and technique challenges to the stakeholder in the usability domain. Here, it is proposed that multiple viewpoints that are driven from UCD method approaches have the potential to reduce or mitigate the problems/issues raised earlier. Hence it is proposed that taking steps to move UCD to a more model driven software engineering approach has the potential to be more effective than taking steps to make Software Engineering more UCD focussed.

## 4.1 Contextual Design

The Contextual Design approach described by Beyer and Holtzblatt [3] is a good candidate for enhancing using model driven principles as it already exhibits language that one might see comfortably in the software design arena but is still a rich user-centered design approach. The method supports the production of a number of artifacts such as: key customer data as the basis for decision making; processes where work is done; interactions using "flow models"; cultural models for capturing intuitive elements of environment; consolidation using affinity diagrams.
These elements are present in a number of models enumerated here and we
also indicate if there is an existing language and notation feature available from
the Unified Modeling Language (UML) [14] available:

**Artifact Model:** produces the key customer data relevant to the system. These data are referred to in other models and also inform the technical architecture element that is part of the User Environment Design stage. The Conceptual Design method does not advocate data modeling approaches explicitly but the strong similarity suggests that UML Class Models would be a strong technique for capturing such data.

**Flow Model:** is an analysis tool that is used to capture communication and coordination between roles. It will typically describe what interactions take place such as request for information, supply of information or an action. In Contextual Design such a model is informal – but it can easily be modeled in UML in a variety of ways. For example flow models have been described by Activity Diagrams with additional features in [?].

**Cultural Model:** is an analysis tool that shows the cultural or political forces in the organization. Issues addressed are forces that may impinge on roles to prevent or modify how work is done. For the purposes of this paper, we view cultural forces having an effect on belief values held by individuals who are participating in a given task.

**Sequence Model:** shows the detailed steps that are performed to accomplish a task. The terminology used in Contextual Design shows strong correspondence to process modeling and can be represented by UML Activity Diagrams. This correlation provides a useful language tool for sharing of information between usability designers and systems designers. For the purposes of this paper we view sequence models as capturing the many alternative workflows that individuals can undertake when directed to perform a given task. The choice between different workflows is partially determined by the belief values held by a given individual. It follows that cultural models have a part to play in influencing workflow choices.

Contextual Design provides a stage that aims to consolidate findings from the analysis. In this stage tools such as Affinity Diagrams for mapping issues across the organization, and Consolidated work models for identifying common strategies are available. These tools help in addressing the problems of multiple types of users, users as designers, confusing what users want and what they truly need because an overall view is possible. Such models could be captured using stereotypes and UML class diagrams.

Similarly, process models described from different user perspectives may be organized in models so that commonality and variability may be explicitly specified. However, no formal language for expressing such variation is described. Interestingly, the method also has elements that are focused on software architecture. These elements produce artifacts that include object models and the functions and structures needed by the re-designed systems expressed as a detailed architectural model.
## 4.2 The Semantics of Cultural Models

We now look at the Cultural model in detail and in particular examine the concepts represented in Cultural models from the perspectives of key components of language design - abstract and concrete syntax and the accompanying semantics. Inspection of examples of cultural models [4]) raises key questions. At first the diagrams appear to convey a significant amount of information. A closer examination requires a full answer to many questions. Many of the diagrams represent Influencers as overlapping circles of different sizes. Consider the following: Is there a significance in the size of circles? Is there significance to the overlaps? What does an overlap mean? Are the length of the arrows important and do they signify anything? Is there a particular style to annotating the circles and arrows? Even if the questions can be answered - and it make take many pages of explanatory text there are still issues of interpretation between users and stakeholders. Generally, these are the same class of problems that modelling design methods have addressed and which led to the standardisation of the Unified Modelling Language (UML) [14] and solutions to these problems boil down to the creation (and agreement) of an abstract syntax (a set of concepts, relationships and well-formed rules), a concrete syntax (typically a graphical notation that supports the concrete syntax), and semantic model for the domain which will allow unambiguous interpretation of instances of concepts from abstract syntax. A model-enhanced version of this diagram would therefore need to be based on a language that is specifically designed to represent cultural forces in an organization - the domain. Such a class of language is often termed a domain specific language (DSL) [6].

# 5 Approach

Our approach to the problem of formalising Cultural model aspects of Contextual Design (CD) is based on the principles of model driven language engineering [7]and through a process of analysis of the problem space, a domain specific language (DSL) is created. The second step is to implement the language using a meta modeling tool or by hand-coding.

## 5.1 Model Driven Language Engineering

A language definition must be provided using a suitable meta-language that can represent the key features of a language (shown in figure 1):

**concrete syntax** is the human-friendly representation of a language. The concrete syntax defines how the language is to be presented on the screen or the page.

**abstract syntax** is a machine friendly representation of language. The abstract syntax defines the information structures that are used to represent the essential features of the language so that they can be processed as data values by a machine without worrying about how they are displayed on the screen or the page.
<figure><img src="image_18.png" /><figcaption>Fig. 1. Model Driven Language Engineering</figcaption></figure>

**syntax mapping** relates the abstract syntax structures to their valid concrete representation. A syntax mapping is used by a tool to link what the user sees to how the language is stored internally.

**semantic domain** is a definition of the things we mean when constructing models in our language. The relationship between syntax and semantics is the same as that between relational database schemas and the tables that conform to them: there are conformance rules and there may be many databases that conform the same schema.

**semantic mapping** are the conformance rules between the syntax and the semantics. For example, the semantic mapping defines the rules by which a database table is considered to be correct with respect to a given schema.

In addition to the elements defined above, a language definition must contain well-formedness rules that define when concrete syntax, abstract syntax and semantic domain elements are valid. These correspond to database rules that, for example, require all column names to be unique.

Simple UML-style class models and associated constraints can be used as a suitable meta-language for representing the language components listed above. The syntax and semantics are represented as independent class diagrams and the mappings are class diagrams that include elements from the appropriate models and define relationships (associations) between the elements. There are other meta-languages available e.g. MOF [12]. Also, Halpin and Morgan's work [16] was used as the meta-modelling language for Archimate - the enterprise architecture modelling language [19].

## 5.2 Tooling

The next step is to provide an implementation of the language, that is, a tool that provides a binding of the various syntactic and semantic models and thus allows users to construct cultural models of the target (or subject) domain. To develop a proof of concept of this toolset, we utilised the meta modelling toolset
MetaEdit+ [17]. Meta Edit+ is a software toolset that supports the design and implementation of domain specific languages. It uses a meta modelling language GOPRR that is broadly similar to one that we used to specify our abstract syntax and has the following concepts: **Graph** specifies one modelling language such as Cultural Model; **Object** describes the basic concepts of a modeling language. Objects are the main elements of the language; Examples include the concept of Force, Role and so on; **Relationship** describes the properties for the objects' connections such as inheritance, call and transition; within the toolset, the relationship mechanism is used to form bindings with objects and roles; **Role** specifies the lines and endpoints of relationships; **Property** defines the attributes which can be used to characterise any of the previous concepts. Properties are of different data types and can be used to link to external concepts. The abstract syntax for Contextual Design was encoded in the GOPRR modelling language within the MetaEdit+ toolset in order to define the concrete syntax and the production of the accompanying tool.

# 6 A DSL for Contextual Design

Contextual Design (CD) involves four different types of model: flow models; sequence models; artifact models; contextual models. It is important that we understand what these models mean in order to use them effectively. This section applies the approach to language design described in section 5.1 to CD.

## 6.1 Abstract Syntax

The abstract syntax is the cornerstone of a language definition. In principle there may be many different concrete syntax models and many different semantics for the same abstract syntax model. This section defines the abstract syntax for each of the main models in the CD modelling language.

<figure><img src="image_19.png" /><figcaption>**Fig. 2.** Flow Models</figcaption></figure>
**Flow Models** Figure 2 shows the abstract syntax of flow-models. The element *Model* is used as the top level container for all CD model elements. A model consists of a collection of roles with flows between them. Each flow represents an interaction between roles and is labelled with both an event that it generates and the artifacts that are involved in the interaction. An example of a well-formedness rule associated with the class *Model* is: every role must have a unique name.

**Artifact Models** Artifact models are equivalent to class models in UML. They describe the elements that are involved in the interactions between roles. In terms of our language definition, we do not need to consider artifacts in any more detail.

<figure><img src="image_20.png" /><figcaption>Fig. 3. Cultural Models</figcaption></figure>

**Cultural Models** Figure 3 shows the abstract syntax of cultural models that represent influences by one role on another. Each influence has a force associated with it from *weak* to *strong*. Each role manages a collection of values that represent personal beliefs. For example an individual might believe that certain types of technology are effective or that the cost associated with using certain processes is very high. In a CD model, *Values* is an assembly of belief value types in the same way that a class has an assembly of attributes. As we shall see, the specific belief values associated with a person who performs the role is defined in the semantics. An *Influence* together with its *Force* defines a condition which must be met by any valid instance of the belief values associated with an any influenced *Role*. A condition is a boolean expression in terms of variables. The effect of applying an influence to a role is to restrict the set of possible belief values that the role can have. A well-formedness rule that applies is *Influence: the set of variable names in the condition must be a sub-set of the value type names associated with the belief values of all influenced roles*.

**Sequence Models** Figure 4 shows the abstract syntax of sequence models. Each role has an interface of activities. Each activity provides a description of what to do when an event is received by the role. Each activity has a number
<figure><img src="image_21.png" /><figcaption>Fig. 4. Sequence Models</figcaption></figure>

of alternative step assemblies (*Steps*) that reflects the options an individual has when responding to a request to perform a task. For example, if an individual is requested to implement a software component they may choose to implement the component in one of a number of programming languages and using any one of a number of development methods. Each individual step in a step-assembly processes some artifacts and must satisfy a collection of belief-values. The idea is that a step cannot be performed by a role unless it is consistent with the beliefs of the particular individual.

An example of a well-formeness rule for sequence models is: *the artifacts associated with a step must be a subset of the artifacts associated with the flow that gave rise to the event.*

## 6.2 Concrete Syntax

The complete abstract syntax for Contextual Design is large. In this paper we have focussed on the Cultural Forces Model as it presents concepts that address areas of the systems design process that are often not captured in software engineering. Consequently we have translated that section of the abstract syntax to the GOPRR meta modelling syntax for Meta-Edit+. The tool capability allows the creation of an concrete syntax - the notations and graphical elements and their binding to the GOPRR representation of the abstract syntax. Diagram 5 shows the abstract syntax for cultural models in MetaEdit+ and figure 6 shows the resulting cultural modelling tool generated from the meta-model and also a partially drawn cultural model of the Social work Domain.

In this diagram example Roles include Student Social Worker. Influencers, those who can assert a force and therefore influence how an activity is performed include: Management, and Academic. Examples of Forces that are brought to bear on a role include the fear of Data security (the loss of data) that was identified during the project deployment. When that force became sufficiently critical, there was a Breakdown (red lightning icon) which resulted in a restriction of the use of the mobile devices. The Toolset for developing and presenting Cultural
<figure><img src="image_22.png" /><figcaption>Fig. 5. GOPRR Meta Model</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Fig. 6. Toolset for Modelling Cultural Forces</figcaption></figure>
Models were generated from the set of abstract concepts described in GOPRR, the concrete syntax developed using the graphics tools available in MetaEdit+ and the bindings and rules for how connections work were declaratively specified again in MetaEdit+

## 6.3 Semantic Domain and Semantic Mapping

<figure><img src="image_24.png" /><figcaption>Fig. 7. Semantic Domain</figcaption></figure>

Figure 7 shows both the elements of the semantic domain (classes suffixed with *Instance*) and elements of the syntax domain together with semantic mapping associations between them (labelled *type*). The semantic domain defines the elements that we are denoting using the syntax models. In this case the semantic elements are essentially sequences of step instances that have arisen from the steps associated with activities in the sequence model. However we cannot associate *any* sequence of steps with a model instance because we must satisfy the semantic mapping constraints that are outlined below:

1. *in every role instance the belief values must satisfy the condition on every influencer of the associated role.*

2. *in every step instance, the condition must be satisfied by the belief values associated with the corresponding role instance.*

3. *a step can only be associated with a role instance where the corresponding role has an incoming event with the same name as the activity giving rise to the steps.*

## 7 Concluding Remarks

The motivating work - the development of a mobile application for a complex domain (Social Work) highlighted that there are potential problems that arise with using co-design and while the core principles of UCD are clearly desirable, the nature of the artifacts that are produced do not transfer to the software
engineering community in a straight forward manner. Thus our experience also
confirms that there is still mileage on the need to converge "on a science of design
through the synthesis of design methodologies" [9]. In particular there is interest
are in how design theories, user centred design approaches in general and their
outputs can be modeled such that method integration with established software
engineering approaches can be more formalized. Hence there a role for model
driven engineering in user centered design and this paper has outlined how one
established UCD approach may be adapted to make it more model driven (and
so artifacts captured using UML modeling tools).

CD models, as defined in the literature, have an informally defined seman-
tics. This limits what can be achieved, especially in terms of tooling to support
CD. This paper has taken a precise meta-modelling approach to the definition
of a language for CD modelling. In doing so, we have defined both the syntax
and (a) semantics for CD. Our semantics defines CD models to denote chains of
steps that arise from interactions between roles in a business context and which
process business artifacts. The semantics reflects the choices that occur in a
business environment that are resolved in terms of belief-systems of the individ-
uals involved; it does this by allowing a single model to denote multiple possible
sequences of steps for each single business activity. The semantics attributes in-
fluencing factors to the ability of individuals in a business to affect belief systems
and thereby influence the way that influenced individuals implement given tasks.

As a result of taking a semantics driven approach to our CD modelling lan-
guage we can now perform analysis of models. For example, it is possible to
determine whether, given a set of influencers on individuals, there are any se-
quences of steps for a given business interaction. Suppose that this is used in a
business that encourages new employees to get advice from established employ-
ees when performing tasks. Our semantics allows us to determine whether there
are particular sets of 'old-hands' whose collective advice would be unhelpful.
Furthermore, we can measure the amount of positive influence that mentoring
is likely to have in terms of the reduction in confusion when staff take on a new
role. For researchers, future projects will likely consider and evaluate further
how such approaches may be used to allow more alignment with the software
engineering model driven architecture paradigm.

References

1. Ra Basnyat, Nick Chozos, and Chris Johnson. Incident and accident investigation techniques to inform model based design of safety critical interactive systems. In *Design, Specification and Verification of Interactive Systems 2005*, pages 51–66. Springer Verlag, 2006.

2. U. Beck. *Risk society: towards a new modernity*. Sage Publications Ltd, 1992.

3. H. Beyer and K. Holtzblatt. *Contextual design: defining customer-centered systems*. Morgan Kaufmann Pub, 1998.

4. H. Beyer and K. Holtzblatt. Contextual design. *interactions*, 6(1):32–42, 1999.

5. G. Bjerknes, P. Ehn, M. Kyng, and K. Nygaard. *Computers and democracy: A Scandinavian challenge*. Gower Pub Co, 1987.
6. T. Clark, A. Evans, and S. Kent. Engineering modelling languages: A precise meta-modelling approach. *Fundamental Approaches to Software Engineering*, pages 242–260, 2002.

7. T. Clark, P. Sammut, and J.S. Willans. Applied metamodelling: a foundation for language driven development. 2008.

8. L.L. Constantine and L.A.D. Lockwood. Usage-centered engineering for Web applications. *IEEE software*, pages 42–50, 2002.

9. G. Fischer. Software engineering themes for the future. In *Proceedings of the 28th international conference on Software engineering*, page 1044. ACM, 2006.

10. Robert France and Bernhard Rumpe. Model-driven development of complex software: A research roadmap. In *FOSE '07: 2007 Future of Software Engineering*, pages 37–54, Washington, DC, USA, 2007. IEEE Computer Society.

11. J.M. Greenbaum and M. Kyng. *Design at work: Cooperative design of computer systems*. L. Erlbaum Associates Inc. Hillsdale, NJ, USA, 1991.

12. Object Management Group. OMG Meta Object Facility, 2010. http://www.omg.org/mof/.

13. Object Management Group. OMG model driven architecture, 2010. http://www.omg.org/mda/.

14. Object Management Group. Unified Modeling Language, 2010. http://www.ulml.org/.

15. J. Gulliksen, B. Goransson, I. Boivie, S. Blomkvist, J. Persson, and Å. Cajander. Key principles for user-centred systems design. *Behaviour & Information Technology*, 22(6):397–409, 2003.

16. T. Halpin and T. Morgan. *Information modeling and relational databases: from conceptual analysis to logical design*. Morgan Kaufmann, 2008.

17. MetaCase Inc. Metaedit+ workbench - build your own domain-specific modeling language, 2009. http://www.metacase.com/mwb/.

18. P. Kruchten. *The rational unified process: an introduction*. Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA, 2000.

19. M. M Lankhorst, H.A H.A. Proper, and J Jonkers. The Anatomy of the ArchiMate Language.

20. P. Marti and L.J. Bannon. Exploring User-Centred Design in Practice: Some Caveats. *Knowledge, Technology & Policy*, 22(1):7–15, 2009.

21. D. Millard, Y. Howard, L. Gilbert, and G. Wills. Co-design and Co-deployment Methodologies for Innovative m-Learning Systems. *Multiplatform E-Learning Systems and Technologies: Mobile Devices for Ubiquitous ICT-Based Education*, 2009.

22. D.A. Norman. Cognitive engineering. *User centered system design*, pages 31–61, 1986.

23. International Standards Organization. *Human-centered design processes for interactive systems*. ISO, 1999.

24. Frank Radeke. *Pattern-driven Model-based User-Interface Development*. 2007.

25. Jean sébastien Sottet, Gaelle Calvary, and Jean marie Favre. Towards mapping and model transformation for consistency of plastic user interfaces. In *The Many Faces of Consistency in Cross-Platform Design Workshop at CHI'2006*, 2006.

26. M. Scaife, Y. Rogers, F. Aldrich, and M. Davies. Designing for or designing with? Informant design for interactive learning environments. In *Proceedings of the SIGCHI conference on Human factors in computing systems*, page 350. ACM, 1997.

27. Jean Vanderdonckt. A mda-compliant environment for developing user interfaces of information systems. In *Proc. of 17 th Conf. on Advanced Information Systems Engineering CAiSE'05*, pages 13–17. Springer-Verlag, 2005.
# An MDE Approach For User Interface Adaptation to
the Context of Use

Wided Bouchelligua<fnref n="1" />,<fnref n="2" />, Adel Mahfoudhi<fnref n="2" />, Lassaad Benammar<fnref n="2" />, Sirine Rebai<fnref n="2" />,
Mourad Abed<fnref n="1" />

**Abstract.** With the advent of new media and the delivery of recent means of communication, associated with the progress of networks, the circumstances of software use, as well as the skills and the preferences of the users who exploit them, are constantly varying. The adaptation of the User Interface (UI) has become a necessity due to the variety of the contexts of use. In this paper, we propose an approach based on models for the generation of adaptive UI. To reach this objective, we have made use of parameterized transformation principle in the framework of the Model Driven Engineering (MDE) for the transformation of the abstract interface into a concrete interface. The parameter of this transformation plays the role of the context of use. The paper develops two parts: meta-models for every constituent of the context of use and the adaptation rules.

**Keywords:** User Interface, Adaptation, Context of use, Model Driven Engineering, adaptation rules.

## 1 Introduction

The technological innovations and the evolution of the means of communication have opened new perspectives to guide the use of the usual applications. Besides, the circumstances of software use have constantly varied following the example of the skills and the preferences of the users who exploit them. This is due to the appearance of new media and the delivery of recent means of communication, associated with the progress of networks. It is not only resources of interaction that can appear and disappear, but also the objectives of the user. The latter is considered as a motive, evolving in a varied environment, according to his needs, to diverse platforms of interaction. That is why, in 1999 Thevenin brought a new concept: the plasticity of

<footnote n="1">¹University of Valenciennes</footnote>

<footnote>LAMIH, UMR CNRS 8530, BP: 311-59304, Valenciennes cedex 9, France</footnote>

<footnote>{wided.bouchelligua, mourad.abed}@univ-valenciennes.fr</footnote>

<footnote n="2">²University of Sfax</footnote>

<footnote>ENIS, CES, Soukra Road km 3.5, B.P: w 3038 Sfax, Tunisia</footnote>

<footnote>adel.mahfoudhi@fss.rnu.tn, {lassaad.benammar, sirine_rebai}@hotmail.com</footnote>
interfaces [21]. The plasticity is defined as the capacity of a user interface to adapt
itself to the context of use which is denoted by the triplet <user, platform,
environment>, while preserving usability.

Several approaches are proposed to make User Interfaces (UI) adaptable to the context of use. According to [18], these approaches are classified into four categories: 1) Translation Interface, 2) Reverse-engineering and migration Interfaces 3) approaches based on the markup languages and 4) model-based approach. The latter is adopted in this work because it has the advantage of applying the adaptation to the context of use of the models, leading to a strong abstraction.

The proposed approach in this paper assures the adaptation of the UI to the context of use. It builds on the concept of transformation parameterized by the context as defined within the framework of the Model Driven Engineering (MDE) [1] [8]. MDE goes beyond the framework of Model Driven Architecture (MDA) [15], which can be summarised in the elaboration of the Platform Independent Models (PIM) and in their transformation into Platform Specific Models (PSM) [1], to cover the methodological aspects. We apply the parameter setting at the level of the transformation of an Abstract User Interface (AUI) into a Concrete User Interface (CUI), whose generation is made on three phases. The first transformation parameterized by the model of adaptation describing the user, gives rise to a first CUI, which in turn is going to feed the second module of transformation. But, the latter will be parameterized by the characteristics of the platform to generate a Concrete User Interface in agreement with the preferences of the user and the properties of the interaction platform. In the last phase, the process of adaptation connected with the environmental context is launched to finish with a plastic Concrete User interface to conform to three dimensions of the context of use.

The remainder of this paper is structured as follows. Section 2 presents a state of
the art on the model-based approaches for the adaptation of the UI. Next, section 3
clarifies the concept of the parameterized transformation in the MDE approach. Then
section 4 describes the proposed approach in terms of context meta-models and
adaptation rules. Section 5 provides a case study illustrating the approach. Finally,
section 6 draws the conclusion and provides perspectives to future research.

## 2 State of the art

This section is limited to the presentation of model-based approaches for UI
adaptation. In fact, the Cameleon reference framework [5] represents an excellent
framework of UI adaptation as it defines four essential stages for the development of
the user interfaces in a pervasive environment (Fig. 1): tasks and concepts, abstract
interface, concrete interface, and final interface.
<figure><img src="image_25.png" /><figcaption>Fig. 1. Cameleon Reference Framework [23].</figcaption></figure>

In this area of research, we can quote the TERESA method [17] that supplies the tasks as a single model, and allows the generation of several interfaces for various platforms. We also cite the Comets (COntext sensitive Multi-target widgETS) [4], which proposes essentially a model for the plastic interactors that can be adapted to the variation of the screen size. Likewise, the UsiXML (User Interface eXtensible Markup Language) [23] [14] approach represents a UI approach of engineering defined according to the Cameleon reference framework. Such an approach describes a context model consisted of three constituents: user, environment and platform. But practically only the variant platform is considered during the UI generation.

Sottet [19] [20] is considered as one of the pioneers to have joined his work to the Model Driven Engineering and the domain of Human Computer Interaction (HCI). His approach makes it possible to show that the concepts of the MDE could be successfully applied to the UI engineering. Sottet [19] proposes meta-models and models transformations to generate adaptable UI, and defines a general context meta-model. Based on the same approach (MDE), Hachani [11] suggests the introduction of the context of use at the tasks level rather than at the interactors level. This approach is distinguished by the definition of the generic rules appropriate to all the contexts of use. However, both approaches lack a detailed description of each constituent of the context of use. As in [20] and [11], we opt for the proposition of a model-based approach and its transformation according to the characteristics of the context. Yet, we seek to detail the context in accordance with three generic meta-models (user meta-model, platform meta-model and environment meta-model).

## 3 MDE Parameterized Transformation

Our objective is to handle the adaptation of the UI to the context of use (platform, environment, user). To do so, our work builds on the parameterized transformations defined by [22]. Vale [22] describes a parameterized transformation within the framework of the model driven engineering for a contextual development. The
methodology proposed by [22] (left of Fig. 2) consists in defining the correspondences "match" between the model of the context and the PIM to define a CPIM (Contextual PIM). Then, an ordinary MDE transformation is used to define the CPSM (Contextual PSM).

<figure><img src="image_26.png" /><figcaption>**Fig. 2.** Parameterized transformation [22].</figcaption></figure>

The correspondences are assured by a parameter setting of the transformation, whose basic principle is to take into consideration the properties of the context during the specification of transformation rules (right of Fig. 2). “*A parameter specifies how arguments are passed into or out of an invocation of a behavioural feature like an operation. The type and multiplicity of a parameter restrict what values can be passed, how many, and whether the values are ordered*” [22].

The use of the parameterized transformations is envisaged with the aim of either improving new features (values, properties, operations) or changing the behaviour of an application. For that purpose, the designer has to specify the parameters which are intended to be inserted during the phase of transformation. In his work, Vale [22] proposes that these parameters are the contextual data, thus and after the transformation, the application will be in harmony with this information passed in parameter.

In the following section, we formulate the principle invoked by Vale, in favour of our approach to implement the notion of plasticity.

# 4 Proposed Approach

The proposed approach consists in generating the user interface automatically, using the parameterized transformation technique of the Model Driven Engineering domain
## 4.1 Overview of the approach

The proposed approach in this paper is shown in Fig. 3. The abstraction levels of the Cameleon framework [5] incorporated in our approach are: abstract user interface and concrete user interface.

In our approach, the Abstract User Interface allows the transition of the specification in the modelling of the abstract components of the interface. In order to describe the Abstract User Interface and the Concrete User interface, we have resorted to the static model of interactions [3]. Aiming at applying a model-to-model transformation, we have refined the static model of the interactions of [3] in the form of two meta-models: the AUI and CUI meta-models. Indeed, AUI meta-model describes the hierarchy of the abstract components corresponding to the logical groups of interactions. The AUI meta-model and transformations rules to obtain the AUI from the task model are detailed in [2].

The Concrete User Interface is deduced from the Abstract User Interface to describe the interface in terms of graphic containers, interactors and navigation objects. We have extended the CUI meta-model presented in [2] [16] to add the vocal components and to associate with every container of the interface a "PersonnalizationService" component containing properties used to specify the presentation of such an object as well as any object being a part of this container. Quoting for example the service "useoflanguage" which can be active if the user prefers a language other than French. If this service is activated, the attribute language allows the specification of the sought language.

The passage to the concrete level has for objective the generation of a plastic interface adapted to the planned context. Our approach facilitates the adaptation of the UI to the user, because the latter is in the centre of all the problem of the UI and everything revolves around him.

The first transformation (T1 in Fig. 3) allows the generation of the first concrete user interface (CUI1 in Fig. 3) adapted to the preferences of the user having received the information suitable to him and echoing them on this intermediate interface.

On the other side of the coin, we are interested in the injection of the characteristics of the platform used to assure the plasticity towards this context. Indeed, we opted for choosing this injection order of the characteristics for multiple reasons. On the one hand, it is around the user that revolves everything and it is his characteristics that are going to impose the choice of the platform. Besides, it is the user who decides about the device on which he even wishes to post the information. Indeed, this variation is going to require the appearance and the disappearance of the other devices of interaction. Furthermore, it is according to his preferences that the modality: graphic, hearing or even olfactive is going to be chosen. Then, in case of change at the level of one of the contextual dimensions, an adaptation is launched to protect the usability [21]. Certainly, the specific properties and the capacity characteristics of the target device have to satisfy the needs of the user. This second transformation (T2 in Fig. 3) adapts the first CUI1 to the characteristics of the platform which is going to welcome the application, from which the second CUI (CUI2 in Fig. 3) results.

So, having fixed and adapted the characteristics of the target platform to his own motivations and intentions, the user has now nothing but to choose the environment
which is going to welcome the application. In fact, this environmental variant has to
be in agreement with the characteristics of the user and the target platform. It is the
profile of the user, defined as being a first entity for the process of adaptation, as well
as these accompanied intentions, naturally, symptomatic of the platform that are going
to determine the environmental aspects. The latter are going to be implemented during
the process of adaptation to succeed in the generation of a plastic UI while taking into
account three speeds of the context. Hence, in the third place, we are going to inject
the environmental properties in the third transformation (T3 in Fig. 3) to have the
interface (CUI3 in Fig. 3).

<figure><img src="image_27.png" /><figcaption>**Fig. 3.** Parameterized transformation for the UI plasticity.</figcaption></figure>

Therefore, the generation of the concrete interface (CUI) is made on three phases.
To do so, we have to establish three meta-models (the user, platform and environment
meta-models), so as to implement the transformation principle to illustrate the process
of adaptation.

**4.2 Context of use Meta-models**

The context is identified by many teams [5] [18] [23] [20] [12] by the triplet `<User Platform, Environment>`. Thus, three categories of contextual information can be distinguished [7]:

- So much information pertaining to the platform (processor, memory, peripheral equipments, connection network, the size of the display screen, and the available interaction tools ...)
- Those relative to the user (his profile, his current activity, his preferences, his habits, his cultural characteristics...)

- The information corresponding to the environment (light, noise, geographical localization ...)

### 4.2.1 User Meta-Model

The user model has to contain information allowing the characterization of the user. Our meta-model (Fig. 4) builds strongly on the work of [10] and [6]. Such information contained is classified in four categories:

* Information staff (the name and the first name of the user, the age, the kind)

* Knowledge (The expertise level of the user in computer science, the expertise level regarding task or manipulated concept)

* Preference (The modality of interaction (graphic, vocal, olfactive, tactile, etc.), police, the character size, colour and the sound volume)

* Capacity (physical (sensory and engine) and cognitive capacities)

<figure><img src="image_28.png" /><figcaption>Fig.4. User Meta-model.</figcaption></figure>

### 4.2.2 Platform Meta-Model

Although most of the work done on plastic UI made adaptation to the platform, it does not provide a complete and detailed meta-model. The existing approaches are limited to its description at a high abstraction level or the description of the display surface of the platform which represents the most used interactional resource in the adaptations made so far. However, the adaptation can be prepared in the presence and absence of the other interaction devices. For example, if we do not have a mouse, we can suggest as a form of adaptation using a vocal interactor where the activation of
the actions will be made vocally. Fig. 5 presents our platform meta-model [16] [2].
Generally, the platform consists of:

* Calculation resources represented in Fig. 5 by the "ComputationalCapacities" class. These resources does not only include the material aspect, such as the memory or processor but also the software aspect as the supported operating system;

* Interaction resources that are the input-output devices represented in our meta-model by the "InteractionDevices" class. We identify two classes of interaction devices: the input devices (InputDevice class in Fig. 5) and the output devices (OutputDevice class in Fig. 5). Certain devices inherit both classes and are thus input/output devices, such as the touch screen.

<figure><img src="image_29.png" /><figcaption>Fig. 5. Platform Meta-model [16].</figcaption></figure>

### 4.2.3 Environment Meta-Model

In this meta-model, illustrated in Fig. 6, we try to cover all the environmental facets of the context susceptible to react directly or indirectly on the interactive system. In fact, we are trying to take into account the maximum of environmental aspects.
Therefore, our meta-model consists of four classes that explain the general characteristics.

* The first class characterizes the ambient environment that surrounds the interactive system "AmbientEnvironment". But with the invasion of the ubiquities computer science, the ambient conditions are changeable from one moment to another. This class inherits three under classes: "ClimaticEnvironment", "LuminousEnvironment" and "SonorousEnvironment".

* The second class composing our meta-model is the class "TemporalEnvironment".

* As for the third class, named "SocialEnvironment", it characterizes the social environment receiving the interactive system. This class is decorated with a single attribute: "atmosphere" of type "Atmosphere" enumeration.

* To specify the place receiving the application, we used the fourth class named "SpatialEnvironment". Indeed, this class gives information about the geographical location of the interactive system.

<figure><img src="image_30.png" /><figcaption>Fig. 6. Environment Meta-model.</figcaption></figure>

## 4.3 Generation process of CUI and adaptation rules

The generation stages of the Concrete User Interface lean strongly on the work of [21] and [14]. The three transformations of the approach are developed with the transformation language Kermeta [13]. The transformation of an *AUI* into a *CUII* (T1 transformation) is implemented by the following four stages:

* Creation of the application: creation of the application in the "ConcreteUserInterface" target model by the "AbstractUserInterface" of the source model;

* Realization of the abstract containers;
* Choice of the interactors;

* Definition of the navigation.

We have developed a set of rules allowing T1 transformation. As an illustration, in what follows, we clarify the stage of interactor's choice. This stage aims at associating the adequate interactor with the abstract component of *AUI*. Such a choice depends on the properties of the abstract component: its type (Input or Output) its nature (Specify, Select, Turn ...) and the user preferences.

The extract of the following code transforms every abstract component of the " CollapsedUIUnit" type into a "UIField" and appeals to the "UIFieldSpecification" method for the choice of the appropriate interactor. In that case, it is a question of executing the interactor's choice for an abstract component of the "Specify" nature.

```
operation createUIField(inputmodel :
    AbstractUserInterface,collapseduiunit: CollapsedUIUnit,
    uiw : UIWindow) is do
    UIFieldSpecification(inputmodel,collapseduiunit,uiw)
end
//UIFieldSpecification
operation UIFieldSpecification( inputmodel :
    AbstractUserInterface,uic : CollapsedUIUnit,uiw :
    UIWindow) is do
    var lnk : Link
    lnk :=ливAllLinks(inputmodel).detect{c|stdio.writeln
        ("link" + c.uicomponent.name)c.uicomponent.name ==
        uic.name}
    //Specify
    var nat : Nature init uic.nature
    var tp : AnnotationType init lnk.uicomponentannot.type
    if (nat == Nature.Specify) then
        createStaticFielduiw,uic,lnk)
        createFieldInuiw,uic,lnk)
    end //rest of code
end
```

Several existing characteristics in the model of the user can have an impact during
the realization of the **AUI**. Certain characteristics have an impact on the choice of the
concrete object of interaction to know the preference of the user in terms of the
modality of communication. The impact is thus expressed in terms of the reshaping of
the interface. The extract of the Kermeta code below illustrates the impact of the
preference modality of communication on the realization.

```
operation transform ( inputModel :
    AbstractUserInterface, paramModel : UserModel) :
    ConcreteUserInterface is do
        AUI2CUI := Trace <UIElement, CUIElement>.new
        AUI2CUI.create
        result := ConcreteUserInterface.new
        var modpref : Modality init
            getPreference(paramModel).modalityOfCommunication
            if (modpref == Modality/graphical) then
```
stdio.writeln("Graphical Modality")
//Graphical treatment
else if modpref == Modality.auditive then
stdio.writeln("Auditive Modality")
//Auditive treatment

end
end

Other characteristics in the model of the user influence the properties of the objects of interactions rather than the choice of concrete object. The extract of the following code shows the function allowing the creation of a service (createServicePerso method). It shows the activation of the two services "useoflanguage" and "useoftooltip" as example. The latter is activated if the user does not have strong computer capacities (computer aptitude).

operation createServicePerso(nameuiw :String,pref : Preference, knl : Knowledge) : PersonnalizationService is do
var srv : PersonnalizationService init
PersonnalizationService.new
srv.name :=nameuiw
if preflanguage != Language.french then
srv.useoflanguage := true
srvlanguage := preflanguage.name
else
srv.useoflanguage := false
end
if knl.computerAptitude != Level.hight then
srv.useoftooltip := true
else
srv.useoftooltip := false
end //rest of code
result := srv
end

The obtained CUI1 is the source model of the second transformation that takes as parameters the characteristics of the platform. We have addressed the impact of the property screen size and inputting/outputting devices of the platform. The following code produces the testing for the required devices of graphical or vocal interaction.

operation transform ( inputModel :
ConcreteUserInterface, paramModel : Plateform) :
ConcreteUserInterface is do
CUI2CUI1 := Trace <CUIElement, CUIElement>.new
CUI2CUI1.create
result := inputModel
var width : Integer init
getScreen(getOutputD(getID(paramModel))).width
var height: Integer init
getScreen(getOutputD(getID(paramModel))).height
getCUIWindow(inputModel).each{uiw1|
if (MouseExist(paramModel) and ScreenExist(paramModel)
and KeyboardExist(paramModel)) or
(TouchPadExist(paramModel) and ScreenExist(paramModel)
and KeyboardExist(paramModel)) or
TouchscreenExist(paramModel) then
//rest of code
else
stdio-writeln("Inexistent Device")
end}
getVocalGroup(inputModel).each{vg|
if VisiocasqueExist(paramModel) or
(MicrophoneExist(paramModel) and ScreenExist(paramModel)
and then
getVocalForm(vg).each{vf|
//rest of code
else
stdio-writeln("Inexistent Device")
end}
end

The third transformation injects the properties of the environment that will host the application. The impact of environment properties does not affect the objects of interaction, but affects the existence or nonexistence of services interface. The following code shows the activation of service "useofbackground".

Rath

getService(inputModel).each{srv |
if (getLuminousEnv(getAmbiantEnv(paramModel)).lightInten
sity == Level.hight) or
(getSocialEnv(paramModel).atmosphere ==
Atmosphere.religious) or
(getSpatialEnv(paramModel).getMetaClass() ==
OutDoorEnvironment and getTemporalEnv(paramModel).time
== Time.daytime) then
srv.useofbackground :=true
srv-background := BackGroundType.light
end

5 Illustrating Example

The case study relates to a credit card request by a customer. This application is
adapted to the context of use. The following scenario illustrates this adaptation on a
precise case. Sarra is connected to the site of the bank to launch her request of credit
card. She has to log in first of all by introducing her user name and password. Then
she has to choose her type (private individual or company). Then, she is asked to
choose the type of card that she seeks to obtain before filling in an information form.
In this case study, the following context of use is assumed:

User={Computer aptitude="lower", font="TimesNewRoman", language="english", color="Red" size="14", Modality of communication ="graphical"},
Platform={iPAQ HX2490 Pocket PC},
Environment={Alone, atmosphere="work", light intensity="low"}.

<figure><img src="image_31.png" /><figcaption>**Fig. 7.** Abstract User Interface for the process of the credit card possession (case of a private individual customer).</figcaption></figure>

Fig. 7 shows the abstract user interface for the process of the credit card possession. This interface contains a "UIGroup" called "Ask for a credit card". This "UIGroup" gives access to two "UIUnitSuit" ("Login" and "Determine private individual form") and "" ("Select customer type").

During the detection of a context change, the system is adapted. The **AUI** adapts
itself first of all to the preferences of the user. In the following figure, we give the
tree-based description of the use of our case study. In front of these characteristics,
the first module of transformation uses as output the **AUI** model (Fig. 7) and the user
model (left of Fig. 8) on which the generic transformations rules are applied, based on
their respective meta-models.
<figure><img src="image_32.png" /><figcaption>**Fig. 8.** (Left) The tree-based description of User model. (Right) Concrete User Interface specific to the user model.</figcaption></figure>

The first module of transformation consists in transforming an XML (Extensible Markup Language) file source obtained from an abstract user interface. This file is automatically generated by our AbstractUserInterface editor developed thanks to the Graphical Modeling Framework (GMF) tool [9] of Eclipse. The result of transformation is an XML file that is in harmony with the CUI meta-model. Right of Fig. 8 produces the **CUII** visualized with our ConcreteUserInterface editor. The realization of the **AUI** is in graphic mode since the user has chosen a modality of graphic communication. A set of personalization services is activated giving as an example the service "Use of tooltip" which results from the fact that the user possesses low computing capacities.

As a concrete example, left of Fig. 9 gives the tree-based description of "iPAQ HX2490 Pocket PC" realized by EMF-based editor. The refinement of the **CUII** taking into account this platform allows the generation of a concrete interface replying on the properties of this platform, as in the example of the value of the screen size (height="320" width="240"). Moreover, the choice of the appropriate interactor is related to the inputting devices that exist in the platform. In this case, we have a touch screen (TouchScreen) and a text input device (TextInputDevice). That is why the concretisation in the graphic form is possible.
<figure><img src="image_33.png" /><figcaption>**Fig. 9.** (Left) The tree-based description of "iPAQ HX2490 Pocket PC". (Right) Concrete User Interface specific to the platform model.</figcaption></figure>

Taking into account the properties of the platform "iPAQ hx2490 Pocket PC" (Left of Fig. 9), the transformation of **CUI1** (Right of Fig. 8) produces a **CUI2** with a remodelling of containers. Right of Fig. 9 presents the visualization of the **CUI2** with our ConcreteUserInterface editor. For readability we have chosen to present only the window "Determine banking information." For the size of the screen "iPAQ hx2490 Pocket PC" and the number of manipulated concepts (>4), the realization of the abstract component "UIGetUnit SELECT Card type" of **AUI** is a "UIUp东西List. A "UIGetUnit SELECT Card type" interactor is added, since the user does not have strong computer Capacities (computer aptitude).

Our case study is situated in a closed environment (inDoorType). As regards the ambient characteristics that specify this type of environment, it will be restored to the intensity of light as well as that of the sound level. This model (Left of Fig. 10) is going to feed the third module of transformation which will lead to the generation of a concrete interface adaptable to the context of use passing through the three elements that define it.

Taking into account the properties of environment (Left of Fig. 10), the transformation of **CUI2** (Right of Fig. 9) producing a **CUI3** with new enabled services, such as the background service whose value has become "gloomy" since the light intensity was low. Right of Fig. 10 produces the visualization of the target **CUI3**.
<figure><img src="image_34.png" /><figcaption>**Fig. 10.** (Left) The tree-based description of environment model. (Right) Concrete User Interface specific to the environment model.</figcaption></figure>

**6 Conclusion**

In this paper, we have presented a methodology for the development of the plastic UI of an Information System. To apply "model to model" transformations, we set up two meta-models: Abstract User Interface meta-model and Concrete User Interface meta-model. The characteristic of the interface adaptation to its context of use was our primordial objective. In order to reach this objective, we proposed three meta-models describing the context of use. Encountered by a new context, a definition of a model for this context will be enough. So, our transformations rules are generic.

We foresee multiple perspectives for our work, which concern the integration of the ergonomic properties in our transformations and the determination of causality between the three components of the context of use.

**References**

1. Bézivin, J., Blay, M., Bouzeghoub, M. et al: Action spécifique CNRS sur l'Ingénierie Dirigée par les Modèles. Rapport de synthèse (2005)

2. Bouchelligua, W., Mahfoudhi, A., Mezhoudi, N., Daassi, O., Abed, M.: User Interfaces Modelling of Workflow Information Systems. In Barjis, J. (Ed.) Enterprise & Organizational Modeling and Simulation. Lecture Notes in Business Information Processing, Volume 63, Springer-Verlag Berlin Heidelberg (2010)
3. Brossard, A., Abed, M., Kolski, C.: Context Awareness and Model Driven Engineering: A multi-level Approach for the Development of Interactive Applications in Public Transportation. In proceedings of 27th European Annual Conference on Human Decision-Making and Manual Control, EAM'08, Delft, Hollande (2008)

4. Calvary, G., Coutaz, J., Dâassi, O., Balme, L., Demeure, A.: Towards a new generation of widgets for supporting software plasticity: the «comet», EHCI-DSVIS'2004, The 9th IFIP Working Conference on Engineering for Human-Computer Interaction Jointly with The 11th International Workshop on Design, Specification and Verification of Interactive Systems, Bastide, R., Palanque, P., Roth, J. (Eds), Lecture Notes in Computer Science 3425, Springer, ISSN 0302-9743, Hamburg, Germany, pp. 306--323 (2004)

5. Calvary, G., Coutaz, J., Thevenin, D. et al : A Unifying Reference Framework for Multi-Target User Interfaces. Interacting with Computers 15, 3, pp. 289--308 (2003)

6. Card, S., Moran, T., Newell, A.: The psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, (1983)

7. Dey, A.: Providing Architectural Support for Building Context-Aware Applications. Thèse de doctorat, Institut Technologique de Géorgie (Georgia Tech), p.170 (2000)

8. Favre, J.-M : Toward a Basic Theory to Model: Model Driven Engineering. Workshop on Software Model Engineering, Wisme' 2004, Lisbonne, Portugal (2004)

9. GMF, Graphical Modeling Framework. www.eclipse.org/gmf

10. Habieb-Mammar, H.: EDPHA: un Environnement de Développement et de Présentation d'Hyperdocuments Adaptatifs. Thèse de doctorat, Institut National des Sciences Appliquées (INSA) de Lyon (2004)

11. Hachani, S., Dupuy-Chessa, S., Front, A. : Une approche générique pour l'adaptation dynamique des IHM au contexte. IHM'09, Grenoble, France (2009)

12. Hariri, M-A., Lepreux, S., Tabary, D., Kolski, C. : Principes et étude de cas d'adaptation d'IHM dans les SI en fonction du contexte d'interaction de l'utilisateur. Ingénierie des Systèmes d'Information (ISI), Networking and Information Systems, 14, pp. 141--162 (2009)

13. Kermeta, Kernel Meta-modeling Framework. http://www.kermeta.org/

14. Limbourg, Q., Vanderdonckt, J.: UsiXML: A User Interface Description Language Supporting Multiple Levels of Independence. In: Matera, M., Comai, S. (eds.): Engineering Advanced Web Applications. Rinton Press, Paramus, pp. 325--338 (2004)

15. MDA, Model Driven Architecture, http://www.omg.org/mda

16. Mezhoudi, N. : Méta-modèles et règles pour la plasticité des IHM, Mémoire de mastère, Institut d'Informatique et de Multimédia, Université de Gabès, Tunisie, (2010)

17. Mori, G., Paternò, F., Santoro, C.: Tool Support for Designing Nomadic Applications. In: Proceedings of the International Conference on Intelligent User Interfaces, pp. 141--148, Miami (2003)

18. Samaan, K., Tarpin-Bernard, F.: Task models and Interaction models in a Multiple User Interfaces generation process. In Proceedings of 3rd International Workshop on TAsk MOdels andDIAgrams for user interface design TAMODIA'2004. Prague, Check Republic, November, ACM, pp. 137-144(2004)

19. Sottet, JS., Calvary, G., Favre, JM., Coutaz, J., Demeure, A., Balme, L.: Towards Model-Driven Engineering of Plastic User Interfaces.In Conference on Model Driven Engineering Languages and Systems (MoDELS'05) satellite proceedings, Springer LNCS, pp. 191—200 (2005)

20. Sottet, JS., Calvary, G., Favre, JM.: Mapping Model: A First Step to Ensure Usability for sustaining User Interface Plasticity. In: Proceedings of the MoDELS'06 Workshop on Model Driven Development of Advanced User Interfaces (2006)

21. Thevenin, D. : Adaptation en Interaction Homme-Machine : Le cas de la Plasticité. Thèse de doctorat, Université Joseph Fourier, Grenoble I, pp. 212 (2001)

22. Vale, S., Hammoudi, S.: Context-aware Model Driven Development by Parameterized Transformation. In proceedings of MDISIS (2008)

23. Vanderdonckt, J.: A MDA-Compliant Environment for Developing User Interfaces of Information Systems. In: Proc. of CAiSE'05, pp. 16--31, Springer-Verlag, Berlin (2005)
<figure><img src="image_35.png" /></figure>
Desktop-to-Mobile Web Adaptation through
Customizable Two-dimensional Semantic Redesign

Fabio Paternò, Giuseppe Zichittella

CNR-ISTI, HIIS Laboratory, Via Moruzzi 1, 56124 Pisa, Italy
{fabio.paterno, giuseppe.zichittella}@isti.cnr.it

**Abstract.** In this paper we present a novel method for desktop-to-mobile adaptation. The solution also supports end-users in customizing multi-device ubiquitous user interfaces. In particular, we describe an algorithm and the corresponding tool support to perform desktop-to-mobile adaptation by exploiting logical user interface descriptions able to capture interaction semantic information indicating the purpose of the interface elements. We also compare our solution with existing tools for similar goals.

**Keywords:** Ubiquitous Applications, Multi-Device Environments, Adaptation.

**Introduction**

One of the main issues in current technological settings is how to design and develop interactive applications that can be accessed through a wide variety of devices (ranging from small watches to very large screens, including various types of smartphones, PDAs and Digital TVs). This is particularly important in Web applications, which are the most common ones.

The vision of ubiquitous computing [16] is that the users operate in intelligent environments, which are aware of users’ needs and able to assist, even proactively, the users in performing their activities and reaching their goals. To this end, one important aspect is the possibility for a user surrounded by multiple devices to freely move about and receive user interfaces adapted to the current context of use.

In current mobile devices various solutions are adopted for accessing Web applications originally developed for desktop systems. Some just cut the page to the display area, thus showing only a limited portion. Others, such as those using the Small Screen Rendering Technique in the Opera mini browser, provide the narrow view in which the content is vertically arranged in order to avoid horizontal scrolling. The most sophisticated solutions are those, such as the Safari browser in the IPhone, which automatically resize the Web page to the screen size and allow the user to zoom in and out through gestures in the touch interface. However, their usability is often low in terms of Web navigation, since users have to make various zoom in and out interactions in order to identify the part of content that they are looking for.

The solutions for such issues can benefit from user interface model-based approaches, in which declarative descriptions of the user interface are used in order to avoid dealing with a plethora of low-level implementation details associated with the