interpretation of data visualization and examples of usability problems that were
found.

However, we found out that in the web environment the browser controls can
influence the list of recorded interactions, e.g. the usage of the back navigation button.
Therefore, the future work in this area should be addition of web browser interactions
recording into the interaction log recorder. With this knowledge, we can introduce
refinements to the user log record and better visualize user interactions as correct or
incorrect. Sometimes it was hard to recognize, what was the object the participant
interacted with. The example is click at "rcbtn101" which means that the user
clicked at the address book. As a future work, we should reflect this and design better
identification of targets both during the interaction recording and data analysis.

## Acknowledgments.

This work has been partly supported by the Ministry of Education, Youth and
Sports of the Czech Republic under the research program LC-06008 (Center for
Computer Graphics) and by the Sucess project – Sun Center of Excellence for
Accessibility & Usability (http://amun.felk.cvut.cz/coe/).

## References

1. Maly, I., Slavik, P.: Towards Visual Analysis of Usability Test Logs. In: Coninx, K., Luyten, K., Schneider, K.A. (eds.) TAMODIA 2006. LNCS, vol. 4385, pp. 25--32, Springer, Heidelberg (2007)

2. Rubin, J.: Handbook of Usability Testing, John Wiley and Sons, New York (1994)

3. Jakob Nielsen's Alertbox, March 19, 2000: Why You Only Need to Test With 5 Users. http://www.useit.com/alertbox/20000319.html

4. Paterno, F., Russino, A., Santoro, C.: Remote evaluation of Mobile Applications. In: Winckler, M., Johnson, H., Palanque, P. (eds.) TAMODIA 2007. LNCS, vol. 4849, pp. 155--168. Springer, Heidelberg (2007)

5. Wurdel, M., Propp, S., Forbrig, P.: HCI-task models and smart environments. In: Forbrig, P., Paterno, F., Pejtersen A.M. (eds) 1st TC 13 Human-Computer Interaction Symposium (HCIS 2008), volume 272 of IFIP International Federation for Information Processing, pp 21--32, Springer, Boston (2008)

6. Paganelli, L., Paterno, F.: Intelligent Analysis of User Interactions with Web Applications. In: Proceedings of ACM IUI'02, pp.111--118, ACM Press, New York (2002)

7. Maly, I., Mikovec, Z., Vystrcil, J.: Interactive Analytical Tool for Usability Analysis of Mobile Indoor Navigation Application. In: 3rd International Conference on Human System Interaction (HSI 2010), pp. 259--266, IEEE, Warsaw (2010)

8. Paganelli, L., Paterno, F.: Intelligent Analysis of User Interactions with Web Applications. In: International Journal of Software Engineering and Knowledge Engineering, vol. 13 (2), pp. 169--189, World Scientific Publishing (2003)

9. Selenium IDE documentation. Accessed on April 8, 2010. http://seleniumhq.org/docs/
# Evaluating Relative Contributions of Various HCI Activities to Usability

Anirudha Joshi¹, NL Sarda¹

¹IIT Bombay, Mumbai 400076 India

anirudha@iitb.ac.in, nls@iitb.ac.in

**Abstract.** Several activities related to human-computer interaction (HCI) design are described in literature. However, it is not clear whether each HCI activity is equally important. We propose a multi-disciplinary framework to organise HCI work in phases, activities, methods, roles, and deliverables. Using regression analyses on data from 50 industry projects, we derive weights for the HCI activities in proportion to the impact they make on usability, and compare these with the recommended and assigned weights. The scores of 4 HCI activities (user studies, user interface design, usability evaluation of the user interface, and development support) have the most impact on the Usability Goals Achievement Metric (UGAM) and account for 58% of variation in it.

**Keywords:** HCI activities, design process, weights.

## 1 Introduction

A human-computer interaction (HCI) design process is made up one or more phases, each of which may consist of one or more HCI activities. Each activity may be associated with one or more methods. A method may require specific skills. An activity may result in a specific deliverable that may be an end in itself, or may be an input for another activity in the HCI design process or the software development process.

For example, usability evaluation is an HCI activity that is a part of almost every HCI design process. Usability evaluation could be performed by several methods such as a think-aloud test, a performance test, a heuristic evaluation, a cognitive walkthrough, or an expert review. Performing each method requires a specific set of skills – e.g. the think-aloud test requires skills in prototyping, qualitative test design, user recruitment, interviewing users, and analysing data. The activity results in deliverables such as usability problems with the design, potential ideas to improve the design, and possibly a decision about the future course of development.

Several HCI activities are described in literature. One or more methods and deliverables are prescribed for each activity. Authors of HCI design processes often express their preference for one method over the other. However, it is not clear whether each HCI activity is equally important. In a specific project, some activities may happen with high level of fidelity, other activities may be cut short, and some activities may not happen at all. Given the context, skipping an HCI activity may have a significant impact on the usability of the product, while skipping or cutting short another activity may only have a marginal impact.
In section 2, we review traditional design literature and HCI literature to articulate the characteristics relevant to design of interactive artefacts. In section 3, we identify 8 HCI activities that we believe are important in any HCI design process and organise them in a multi-disciplinary framework along with their associated methods, roles, and deliverables. In section 4, we propose a method to express the relative importance of these activities. In section 5, we describe a study that we conducted with 50 industrial projects in India to arrive at the relative importance of these activities empirically. In section 6, we present our conclusions.

# 2 Design Activities

## 2.1 Activities in Traditional Design Process

Archer defines design as a goal-driven problem-solving activity [1]. According to Jones [2], the effect of designing is to initiate a change in man-made things that in turn affect the manufacturers of those products, the distributors, the purchasers, the users, and ultimately the society. An important job of the designer is to predict each of those behaviours and responses at each stage in the life of the product.

One of the ways to understand design is to chart the design process [3]. Several authors agree that at its bare bones, a systematic design process comprises of three fundamental activities [1], [2], [3], [4]:

* Analyse the user needs, the problems and the opportunities to identify the goals and the constraints

* Synthesise alternative solutions

* Evaluate them against goals and redesign the product where necessary.

Authors also agree that the design processes are iterative. Problems found with the proposed design at the time of evaluation are fixed in a new design solution and this is done until the most appropriate solution is found. As iterations progress, the design also moves from generic to detailed. Designers have evolved many methods to carry out these activities. The main effect of the design methods is to externalise what good designers do intuitively to allow design of complex and innovative systems that might be beyond the experience of any one designer.

The need for expanding upon the design brief itself before converging to a solution has been expressed by several authors, including Jones [2] and Laseau [5]. Jones broadly divides design methods into three categories that correspond to the three stages of design – divergence, transformation, and convergence [2].

Divergence refers to the act of extending the boundary of the design situation to have a large search space in which to seek a solution. The aim of divergent search is to restate the original brief while identifying the features of the design situation that will permit a valuable and feasible degree of change. Key characteristics of the divergence stage are its tentativeness and instability. The objectives, the problem boundary, and the sponsor's brief are unstable, and evolve during this stage and evaluation is deliberately deferred so that nothing relevant is disregarded. Design methods related to this stage often require both rational and intuitive actions, and many of them require "legwork rather than armchair speculation"[2].
Transformation is the creative and the most interesting step of design when the objectives, the brief, and the problem boundaries are fixed, the critical variables are identified, the constraints are recognised, and the opportunities are taken. Jones warns that this could also be the stage where big blunders are made, and where experience and sound judgement are necessary. Design methods for searching for new ideas (such as brainstorming and synectics), and design methods to explore the problem structure (such as mind mapping, interaction matrix, and affinity) enable this transformation. Jones calls many of these methods as “black-box methods” as these depend on the chief designer’s creativity and intuition [2].

In convergence, the designer’s aim is to reduce the secondary uncertainties rapidly so that an optimum solution can be arrived at with minimal effort. During this stage, the designer is working with the most details in design and if he does not converge fast, the number of alternatives available can explode. Design methods related to convergence stage are related to evaluation, measurement, and analysis. Jones calls these as “glass-box methods” as these are very rational and analytical [2].

## 2.2 Activities in HCI Design Process

Many authors have prescribed process models specifically for the design of interactive products. Several of these (particularly the early authors) came from backgrounds in psychology, and their process models reflect a stronger emphasis on analysis, usability evaluation, and convergent thinking. Nevertheless, there are many overlaps with the traditional design processes, particularly in the later literature.

The basic ideas for design of interactive systems were already articulated by the 1980s. Gould and Lewis recommended three “principles” of design, which easily translate into the steps of a process: early focus on users and tasks, empirical measurement of user performance on prototypes, and iterative design to fix problems found during usability tests “as they will be” [6]. They also acknowledged the importance of the process to ensure meeting usability goals.

More detailed process models have been proposed by other authors. Nielsen suggests a 11-stage usability engineering lifecycle model [7]. Kreitzberg identifies a 6-stage design methodology [8]. Dix et al. relate the HCI design process to software development lifecycle [9]. Preece et al. emphasise the need to look “beyond HCI” into interaction design [10], [11].

Contextual Design process developed by Beyer and Holtzblatt explicitly brings in divergence and transformation in addition to convergence [12]. Divergence is enabled mainly by the technique of contextual inquiry, an interview technique that draws upon ethnography and allows designers to gain deep understanding of users’ tasks, roles, artefacts, environment, and culture. Transformation is brought in by consolidating findings across users through techniques such as affinity and redesign of users’ work with a vision of the design that drives changes to the organisational work practice.

Cooper and Riemann’s goal-directed design process is driven by roles such as managers, designers, programmers, software testers and usability testers [13]. Their design process consists of steps that reflect divergence, transformation, and convergence: Research users and the domain; model users and use contexts (personas and their goals); define requirements of users, business, and technology; create a
framework to define the design structure and flow through scenarios; refine the
framework; design the interface details; and validate them.

Mayhew brings the perspective of an external usability consultant to the product
development process [14]. Mayhew suggests a variety of techniques to carry out each
task, but her approach is open and flexible – one may substitute a quicker / cheaper
technique to do a task, but each task must be done. (Our approach to activity and
method is similar to Mayhew’s approach to task and technique). Garrett divides users
experience of a website in five layers – surface, skeleton, structure, scope, and
strategy [15]. Garrett’s model of user experience is not a process model in itself, but it
has important implications for the process. Decisions at lower layers affect the
choices available at the higher layers. Therefore, a strategic decision will ripple
through the scope, the structure, the skeleton, and the surface. Similarly, changes in
the scope will affect the structure, the skeleton, and the surface. Therefore, one needs
to consider as many alternatives as possible before freezing upon the strategy and the
scope.

Gulliksen et al. review HCI literature and list key principles of user centred design [16]. One of the principles they emphasise is that the design should be holistic, considering all aspects including impact of design on the users’ work, on the organisation, roles, etc. All parts of the product (task organisation, user interface, online help, user training, health and safety aspects etc.) should be influenced by common design thinking.

3 Framework for HCI Design

By combining the essential characteristics of the various processes discussed above, we propose a process framework for HCI design. In our framework, we prescribe 8 HCI activities, and associate them with typical methods, and deliverables. We organise the activities in phases, which we describe in terms of four questions derived from [12]: *What matters? How should we respond? How should the design be detailed? How are we doing?*

**Figure 1** captures the phases, activities, methods, skills, and deliverables in a visual form. **Table 1** summarises the same in a tabular view. Below, we describe this framework in detail. The framework is a flexible way of understanding and communicating the work of HCI designers in different contexts. Our objective is not to prescribe a one-size-fits-all HCI design process, but rather to articulate the typical HCI activities within which several methods and deliverables can be assimilated. Not all activities or methods may be essential in each instance of the process.
<figure><img src="image_80.png" /><figcaption>**Figure 1:** Proposed framework for HCI design process.</figcaption></figure>

## 3.1 What matters?

It is not only about what is “required” by someone. “What matters?” is a broader question and asks the design team to look at the problem at hand as holistically as possible. This question is answered through divergent thinking, looking beyond what had been specified in the design brief. The HCI activity associated with this phase is user studies, user modelling, and market analysis (activity 1 in **Table 1**). To understand the key concerns of the users, the stakeholders, the domain, and the context deeply, the team uses methods such as stakeholder interviews, contextual user interviews, focus groups, field observations, log analyses etc. The team may also study related issues such as the environmental, social, or cultural aspects. Most teams would do a benchmark analysis of competitive products. This is a very multi-disciplinary phase where ethnographers, business analysts, domain experts, client / business stakeholders, designers, and potential users are involved. At the end of this phase, the team gets a good understanding of users’ needs, problems, goals, and constraints. They also have a good understanding of the design opportunities. The phase ends with identifying product goals, including usability goals.
## 3.2 How should we respond?

This is a holistic question as well. Now the design team is not just describing the situation but also transforming the problem space so that one or a few solutions become evident. The team begins with ideation (activity 2 in **Table 1**). With their creativity, but also using a range of ideation techniques such as brainstorming, synectics, participatory design, quality function deployment, and TRIZ (theory of inventor's problem solving) the team comes up with a range of design ideas that solve the problems and realise the opportunities.

The ideas could be wild and divergent to begin with, but eventually the team reaches a coherent understanding and articulation of the context and creates a meaningful, holistic response – a high-level product definition (activity 3 in **Table 1**). In interaction design, the product definition is often expressed through personas and scenarios. Sometimes, low-fidelity prototypes are created to support the scenarios. If design involves a new hardware, the form factor is modelled. If it involves software, wireframes of the screens are created. Buxton calls these techniques as ‘sketching user experiences’ [17].

In this phase, a multi-disciplinary team is involved, including designers, business analysts, engineers, and client / business stakeholders. If the method of participatory design is used, users are also involved. The product goals guide the team in this phase, but the product goals are also reviewed.

The first loop in the framework is the *feasibility* loop that occurs just after the product definition. Here business and technical feasibility of the proposed product definition are assessed. If competing product definitions are still in contention, a choice is made. If none of the proposed product definitions is found to be feasible, the team goes back and think of more alternatives. At this stage, the design team may do a formative evaluation of the product definition (activity 4 in **Table 1**), possibly by lightweight methods such as a heuristic evaluation or a cognitive walkthrough. The product definition would be refined to fix any problems found. At the end of this activity, product goals are finalised and technically and financially feasible product definition is agreed.

## 3.3 How should the design be detailed?

Once a feasible product definition is agreed upon, the detailed user interface is designed (activity 5 in **Table 1**). This activity completes the transformation and initiates the convergence. Designers explore the details of the user interfaces such as labels, icons, and behaviour of widgets. The text is written. Information is visualised. Visual elements such as typography, colours, fonts, and layouts are designed. Product form is finalised. Design decisions that seem particularly risky are prototyped first so that feedback on these can be sought early. This activity is primarily the designers’ responsibility, though truly innovative designs may require collaboration between design, technology, and business. The output of this phase is one or more prototypes capturing and representing the design decisions.
## 3.4 How are we doing?

In this phase, the team seeks to converge to a usable solution quickly. The purpose of creating a prototype is to evaluate it. As it may happen, the initial design decisions do not fit all the users' needs. When a prototype is ready, a formative usability evaluation is done against the usability goals to identify potential problems with the design (activity 6 in **Table 1**). Card sorts and think-aloud tests are the most preferred method of evaluation at this stage, but other methods may also be used. Usability evaluators do the evaluations, though designers may also participate to get a first-hand feedback. The evaluation generates a list of problems and design ideas to fix them.

The second loop in the framework is that of redesign. As Gould and Lewis, interactive systems are particularly prone to having problems in the early designs [6]. After a round of evaluation, problems are fed back and products are redesigned until an acceptable solution is found. The fidelity of the prototypes keeps increasing through the iterations as more details are added. This cycle of design, prototype, evaluate, redesign needs to be tight, quick and consume as few resources as possible.

Changes to the design inevitably happen during software development. Some changes are inadvertent slip-ups that need to be corrected (e.g. an accidental change of typeface, colour, or layout). Other changes have to be made because the original design was not feasible. Yet other set of changes happen because there is a change in requirements or change in technology platform. In all cases, ongoing collaboration between the design and engineering teams is important during software development – we call this development support (activity 7 in **Table 1**).

When an early version of the production code becomes available, it is a good idea to do a summative usability evaluation against the usability goals (activity 8 in **Table 1**). Often summative evaluation is done in a lab-based quantitative performance test. In some cases, it may be done by deploying the product in the field. Preferably, a summative evaluation is done by an external evaluator. The main outcome of a summative evaluation is (hopefully) a usability approval. A set of metrics could also emerge. Though summative evaluation is not supposed to affect the design, if serious usability problems are found, these ought to be fixed before release.

<table><caption>**Table 1:** A multi-disciplinary framework for the HCI design process. Asterisk (*) denotes necessary deliverables.</caption><thead><tr><td>Phases</td><td>Disciplines</td><td>HCI Activities</td><td>Methods</td><td>Deliverables</td></tr></thead><tbody><tr><td>What matters?</td><td>Ethnographers, business analysts, domain experts, client / business stakeholders, designers, users</td><td>1. User studies, user modelling, market analysis</td><td>Stakeholder interviews<br>Contextual inquiry<br>Focus groups<br>Competitive product analysis</td><td>Analysis of individual interviews<br>User models such as affinity, work models, mind-maps, personas<br>User needs, problems, goals and constraints*<br>Opportunities for design interventions<br>Product goals (including usability goals)*</td></tr></tbody></table>
<table><thead><tr><th>Phases</th><th>Disciplines</th><th>HCI Activities</th><th>Methods</th><th>Deliverables</th></tr></thead><tbody><tr><td rowspan="2">How should we respond?</td><td rowspan="2">Designers, business analysts, engineers, client / business stakeholders, ethnographers, users</td><td>2. Ideation</td><td>Brainstorming<br>Participatory design<br>TRIZ<br>QFD</td><td>Design ideas</td></tr><tr><td>3. Product definition</td><td>Interaction design<br>Information architecture</td><td>High-level use scenarios, storyboards<br>Low fidelity prototypes, wireframes of software, foam models of hardware<br>Business model<br>Strategy, scope and structure of Garrett's model</td></tr><tr><td>Feasibility</td><td>Engineers, client / business stakeholders, usability experts</td><td>4. Formative usability evaluation 1 and refinement</td><td>Heuristic evaluation</td><td>Refined and approved product definition and product goals*<br>Technology feasibility approval*<br>Business feasibility approval*</td></tr><tr><td>How should the design be detailed?</td><td>Designers, engineers</td><td>5. Design detailing</td><td>Interface design<br>Information design<br>Navigation design<br>Visual design<br>Product form design</td><td>Medium to high fidelity UI prototypes through iterations<br>Structure, skeleton and surface of Garrett's model</td></tr><tr><td rowspan="2">How are we doing?</td><td rowspan="2">Usability experts, designers, users</td><td>6. Formative usability evaluation 2 and...</td><td>Heuristic evaluation<br>Cognitive walkthrough<br>Think aloud test<br>Card sorting</td><td>Usability problems<br>Metrics</td></tr><tr><td>... refinement</td><td>Same as in design detailing</td><td>Refined, detailed UI prototypes*<br>UI specification*</td></tr><tr><td rowspan="2">Dev. support</td><td>Designers, usability experts</td><td>7. Development support</td><td>Reviews during development</td><td>Minor tweaks</td></tr><tr><td>Usability experts, users</td><td>8. Summative usability evaluation 3</td><td>Usability performance test<br>Field trials</td><td>Usability approval*<br>Metrics</td></tr></tbody></table>

## 4 Recommended Weights for HCI Activities

The HCI activities in our framework must be integrated with the software engineering process model in use, so that they are applied in the practice of software development. Further, each activity may not be equally important in all situations. The importance of an activity would depend on the nature of the product, the context, and the experience of the team. In this section, we recommend the importance to be assigned to each HCI activity in our framework appropriate for typical contexts. However, note that the importance may vary in specific cases (some examples of which we point out). We express the importance of an activity by assigning it a weight on the scale of 0-5, where 0 indicates that the activity is not relevant, 1 indicates the activity is
somewhat relevant, 2 indicates the activity is of typical importance, 3 indicates the activity is more important than usual, 4 indicates that the activity is very important, and 5 indicates that the activity is extremely important. Expressing the importance of these activities in this manner helps in direct evaluation of process metrics, as we describe in [18] and [19].

We will demonstrate the use of this framework with the waterfall model of software engineering. Despite criticisms, the waterfall model is still popular in the industry. In a survey of 200 practitioners, Neill and Leplante reported that the waterfall model was the most dominant and 35% of the practitioners claim using it [20]. In our experience, the waterfall model is used even more extensively in the Indian software industry.

To integrate our framework with the phases of the waterfall model, we suggest that the Communication phase of the waterfall model should include activities 1-4 of our framework, the Modelling phase should include activities 5-6 and the Construction phase should include activities 7-8 [21]. **Table 2** lists our weight recommendations for each HCI activity when integrated with the waterfall model in this manner. Below, we describe our rationale for these weights.

<table><caption>**Table 2:** Initial weights recommendations for HCI activities in the waterfall model.</caption><thead><tr><td>HCI Activity</td><td>Recommended weights</td></tr></thead><tbody><tr><td>1. User studies, user modelling, competitive product analysis</td><td>3 - 4</td></tr><tr><td>2. Ideation with a multidisciplinary team</td><td>2</td></tr><tr><td>3. Product definition</td><td>1 - 3</td></tr><tr><td>4. Usability evaluation 1 of the product definition and refinement</td><td>1 - 3</td></tr><tr><td>5. User interface prototyping</td><td>4 - 5</td></tr><tr><td>6. Usability evaluation 2 of the user interface and refinement</td><td>4 - 5</td></tr><tr><td>7. Development support: ongoing reviews by usability team during development</td><td>3</td></tr><tr><td>8. Usability evaluation 3 of an early version</td><td>1 - 3</td></tr></tbody></table>

In the beginning of a project, it is very important to understand the context of the user and the market scenario. Hence, the activities related to **user studies** and competitive product analysis is recommended a weight of 3 to 4. The weight can increase if the team is especially unfamiliar with the domain and the context, and can decrease if the team is very familiar with the domain and the context.

**Ideation** is an important activity. However, doing ideation formally as an independent activity may not be as important as generating ideas. Since user studies may also generate many ideas, the importance of explicit ideation is somewhat less. We therefore give it a weight of 2. However, extensive user studies may not be done if the product is not based on contextual data but on ideas, (for example, a toy, or an interactive installation). In such cases, the weight of ideation will go up.

**Product definition** is given a weight of 1 to 3 because we feel this activity can vary in importance. In situations where the product is very innovative or particularly unpredictable, the weight of this activity can go up. On the other hand, if the product is very predictable and what needs to be done is clearly understood by all, the weight can go down.
**Detailed UI prototyping** is the crux of the HCI design process as the main deliverables of HCI professionals come forth from it. This activity is therefore recommended a weight of 4 to 5.

In our framework, we identify three occasions where usability evaluation can be done – just after the product definition, after detailing out the user interface, and after an initial version of the working product becomes available. Of these, the first two are formative (aimed at improving the design), while the last one is summative (aimed at ensuring that all goals have been met). The formative evaluations are important as they directly affect the design. Between the two formative evaluations, we expect **usability evaluation 2 of the user interface** to be more important in many contexts as it will evaluate the design with many of its details in place. This evaluation is therefore recommended a weight of 4 to 5. We assume that if this formative evaluation was done well, the importance of doing the other two usability evaluations will be less.

**Usability evaluation 1 of the product definition** will usually have to be done on a very low fidelity prototype under very tight deadlines. Hence, we recommend a low weight for this step. In practice, the situations may vary somewhat. There may be opportunities (e.g. high-fidelity prototype was available early) and reasons (e.g. to demonstrate ideas to investors) to give more importance to the first formative evaluation. In this case, the weight of this usability evaluation can be increased and correspondingly, the weight for the next usability evaluation can be decreased.

**Usability evaluation 3 of an early release** is a summative evaluation and is expected to have little impact on design. Hence, it was also assigned a weight of 1. However, in projects where user is expected to do critical tasks, this step will gain weight of up to 3.

Finally, we reckon that a lot depends on the continued contact between the HCI professionals and the development teams after the activity of detailed UI prototyping has been completed. Unanticipated UI changes may arise late in the project. In many companies, the HCI professionals are a shared resource and they keep moving from one project to the next before the earlier project is over. To emphasise the importance of **development support** and reviews of design changes during software development, we assign this step a weight of 3.

# 5 Validating Recommended Weights

## 5.1 Method

We derived the relative contributions of HCI activities in our framework to usability (and validated the weights proposed in section 4) with the help of simple linear regressions of each activity and a stepwise multiple linear regression of all activities on the usability of products in real-life industrial projects.

As a measure of the usability, we selected Usability Goals Achievement Metric (UGAM), a product metric that measures the extent to which the design achieves the usability goals. To calculate UGAM, high-level user experience goals are broken down into detailed, measurable goal parameters. For example, parameters for the
high-level goal of learnability could be: options / data / information should be easy to find, user should take little time to learn, user should be able to learn on his own, the product should be consistent with its earlier version, etc. Each goal parameter is assigned a weight between 0-5. During a usability evaluation, each goal parameter is assigned a score between 0-100. UGAM is the sum of the weighted average of the scores,

$$UGAM = \frac{\sum W_p \times S_p}{\sum W_p}$$

where $W_p$ is the weight of the goal parameter p and $S_p$ is its score. UGAM is described in more detail in [18] and [19]. Goals and goal parameters are described in more detail in [22].

HCI professionals working in the Indian IT industry were invited to participate in the study. Participants were taught the method of calculating UGAM. They were also walked through the HCI activities in our process framework. First, participants were asked to calculate UGAM scores of the products delivered by their projects. Participants were then asked to assign a weight to each HCI activity based on their judgement of the importance of that activity in the context of their project. While they were shown the recommended weights described above, they were also given the freedom to assign a different score if they wished.

Finally, participants were asked to assign a score to each HCI activity from 0 to 100, where 100 represents the best case situation i.e. the activity was done in the best possible manner, with the highest fidelity, in the most appropriate phase of software development and with the best possible deliverables; 75 represents that the activity was somewhat toned down, but was still well-timed and well-executed; 50 represents an undecided state where the activity was done with some shortcuts or perhaps was not timed well; 25 represents that the activity was done with many shortcomings; and 0 represents the worst case situation where the activity was not done at all.

To help participants assign a score to each activity, we came up with detailed guidelines for evaluating each activity. For example, following are the guidelines for the activity 1 – user studies, user modelling, and competitive product analysis:

1. Both organizational data gathering and user studies are done before requirements are finalized.

2. User studies are done in the context of the users by the method of contextual inquiry.

3. User studies are done with at least 20 users in each profile.

4. User studies are done by people with experience in user studies in a similar domain of at least 2 projects.

5. The findings including user problems, goals, opportunities, and constraints are analyzed, documented, and presented in an established user modelling methodology such as personas, work models, affinity diagram, etc.

6. Competitive / similar products and earlier versions of the products are evaluated for potential usability problems, at least by using discount usability evaluation methods such as heuristic evaluation, and are benchmarked.

7. User experience goals are explicitly agreed upon before finalizing requirements. 100 = All the above are true, the activity was performed exceptionally well, 75 = At least five of the above are true, including point 7, or all the above are true, but point 3 had fewer than 20 users per profile, the activity was performed reasonably
well, 50 = At least three of the above are true, including point 7, the activity was done with some shortcuts and / or perhaps was not timed well, 25 = Only two of the above are true, the activity was done poorly with many shortcomings, 0 = None of the above are true, the activity was not done.

Detailed guidelines for all activities are available online [23].

## 5.2 Weights Assigned by Participants

A total of 36 participants submitted 50 projects (some participants submitted more than one project). The HCI related experience of participants was between 1-7 years. The participants came from a wide variety of companies including large contracted software development companies, smaller contracted software development companies, multi-national companies with large product development centres in India, one large, internationally popular internet company, and five smaller product development companies. Only the projects following the waterfall model were used for the analyses presented in this paper.

**Table 3** lists the averages of weights actually assigned by participants for HCI activities and their standard deviations. Participants do not seem to have deviated substantially from our recommendations.

<table><caption>**Table 3:** Initial recommendations for weights of the HCI activities and the average and the standard deviation of weights actually assigned by participants to those HCI activities (N = 50).</caption><thead><tr><td>HCI Activity</td><td>Recommended weights</td><td>Assigned weights average</td><td>Assigned weights SD</td></tr></thead><tbody><tr><td>1. User studies, user modelling...</td><td>3 - 4</td><td>3.7</td><td>0.8</td></tr><tr><td>2. Ideation with a multidisciplinary team</td><td>2</td><td>2.5</td><td>0.7</td></tr><tr><td>3. Product definition</td><td>1 - 3</td><td>3.1</td><td>0.7</td></tr><tr><td>4. Usability evaluation 1 of product definition...</td><td>1 - 3</td><td>2.0</td><td>1.1</td></tr><tr><td>5. User interface prototyping</td><td>4 - 5</td><td>4.5</td><td>0.6</td></tr><tr><td>6. Usability evaluation 2 of the user interface...</td><td>4 - 5</td><td>3.8</td><td>0.8</td></tr><tr><td>7. Development support...</td><td>3</td><td>3.2</td><td>0.8</td></tr><tr><td>8. Usability evaluation 3 of an early version...</td><td>1 - 3</td><td>1.9</td><td>1.0</td></tr></tbody></table>

## 5.3 Weights Derived from Regression Analysis

The score of each HCI activity is a measure of the fidelity of that HCI activity. UGAM is a measure of usability goal achievement in the project. The UGAM score is arrived at independently of the scores of HCI activities. If we can find the relative effect of the scores of HCI activities on the UGAM scores, this could be a way of evaluating the impact of HCI activities on the usability.

Separate simple linear regressions were performed assuming the scores of each of the eight HCI activities to be the predictor variables and UGAM to be the criterion variable (Table 4). In case of each HCI activity, a significant model emerged and the activity score had a positive significant Pearson's correlation with UGAM (0.56 > R >
0.33, 0.32 > R² > 0.11, 0.30 > adjusted R² > 0.09, 22.399 > F > 5.796, p <= 0.02, two-tailed). All coefficients were positive. All lower bounds of the 95% confidence intervals of the coefficients were also positive.

We can conclude that all HCI activities recommended in **Table 2** affect UGAM positively. The scores of the HCI activities seem to be affecting the UGAM scores to varying degrees – some HCI activities have a larger effect on UGAM than others. The strongest correlations, largest adjusted R² values, and largest coefficients were observed for the HCI activities of user interface prototyping, usability evaluation of the user interface and refinement, development support, and user studies, user modelling, competitive product analysis. This justifies our 3+ weight recommendations for these activities (Table 2) and also the 3+ average weight assigned by participants (Table 3).

The adjusted R² value in a simple linear regression represents the extent to which a predictor variable affects the criterion variable. We could possibly assign weights to the HCI activities derived in proportion to the adjusted R² values we show below in column 4 of Table 6.

<table><caption>**Table 4:** Summary of simple linear regressions on UGAM as criterion variable and the scores of individual HCI activities as predictor variables on merged project scores (N = 50). The top four correlating activities have been highlighted.</caption><thead><tr><th rowspan="2"></th><th colspan="5">Model</th><th colspan="3">95% conf. interval for B</th></tr><tr><th>R</th><th>R<sup>2</sup></th><th>Adj. R<sup>2</sup></th><th>F</th><th>Sig.</th><th>B</th><th>t</th><th>Sig.</th></tr></thead><tbody><tr><td>User studies</td><td>0.445</td><td>0.207</td><td>0.190</td><td>12.517</td><td>0.001</td><td>0.221</td><td>3.538</td><td>0.001</td><td>0.095</td><td>0.346</td></tr><tr><td>Ideation</td><td>0.384</td><td>0.148</td><td>0.130</td><td>8.326</td><td>0.006</td><td>0.190</td><td>2.886</td><td>0.006</td><td>0.057</td><td>0.322</td></tr><tr><td>Prod Def</td><td>0.406</td><td>0.165</td><td>0.148</td><td>9.481</td><td>0.003</td><td>0.227</td><td>3.079</td><td>0.003</td><td>0.079</td><td>0.375</td></tr><tr><td>UE 1</td><td>0.351</td><td>0.123</td><td>0.105</td><td>6.748</td><td>0.012</td><td>0.162</td><td>2.598</td><td>0.012</td><td>0.037</td><td>0.287</td></tr><tr><td>UI Proto</td><td>0.564</td><td>0.318</td><td>0.304</td><td>22.399</td><td>0.000</td><td>0.299</td><td>4.733</td><td>0.000</td><td>0.172</td><td>0.426</td></tr><tr><td>UE 2</td><td>0.534</td><td>0.285</td><td>0.270</td><td>19.126</td><td>0.000</td><td>0.249</td><td>4.373</td><td>0.000</td><td>0.134</td><td>0.363</td></tr><tr><td>Dev Support</td><td>0.532</td><td>0.283</td><td>0.268</td><td>18.967</td><td>0.000</td><td>0.216</td><td>4.355</td><td>0.000</td><td>0.116</td><td>0.315</td></tr><tr><td>UE 3</td><td>0.328</td><td>0.108</td><td>0.089</td><td>5.796</td><td>0.020</td><td>0.134</td><td>2.407</td><td>0.020</td><td>0.022</td><td>0.246</td></tr></tbody></table>

Using the stepwise method, a multiple regression was performed assuming the scores of the eight recommended HCI activities as predictor variables and UGAM as the criterion variable. The most significant model returned these values: R = 0.784, R² = 0.614, adjusted R² = 0.580, F = 8.533, p < 0.005. The four HCI activities identified above also emerged as significant predictors in this model (Table 5). The scores on these four HCI activities predicted 58% of variation in UGAM (adjusted R² = 0.580). These four HCI activities had a positive, significant coefficient (p <= 0.023) and the lower bound of the 95% confidence interval for all coefficients was positive. The variance inflation factors (VIFs) of all predictor variables are well below 4, indicating that there is no multi-collinearity among the predictor variables. This implies that the assumption that the HCI activity scores are independent variables was acceptable for the purpose of the stepwise multiple regression.
<table><caption>**Table 5:** The most significant model in the SPSS output of stepwise multiple linear regression on UGAM as criterion variable and the ratings of HCI activities as predictor variables (n = 50).</caption><thead><tr><th rowspan="2">R</th><th rowspan="2">R<sup>2</sup></th><th rowspan="2">Adj. R<sup>2</sup></th><th rowspan="2">Std. Error of the Estimate</th><th colspan="5">Change Statistics</th></tr><tr><th>R Square Change</th><th>F Change</th><th>df1</th><th>df2</th><th>Sig. F Change</th></tr></thead><tbody><tr><td>0.784</td><td>0.614</td><td>0.580</td><td>7.702</td><td>0.073</td><td>8.533</td><td>1</td><td>45</td><td>0.005</td></tr></tbody></table>

<table><thead><tr><th rowspan="2"></th><th colspan="2">Unstandardised Coefficients</th><th rowspan="2">Standardised coefficients</th><th rowspan="2">t</th><th rowspan="2">Sig.</th><th colspan="2">95% Confidence Interval for B</th><th rowspan="2">Collinearity Statistics</th></tr><tr><th>B</th><th>Std. Error</th><th>L. Bound</th><th>U. Bound</th><th>VIF</th></tr></thead><tbody><tr><td>(Constant)</td><td>33.794</td><td>4.024</td><td></td><td>8.398</td><td>0.000</td><td>25.690</td><td>41.889</td><td></td></tr><tr><td>Usability Eval 2</td><td>0.154</td><td>0.048</td><td>0.332</td><td>3.208</td><td>0.002</td><td>0.057</td><td>0.250</td><td>1.247</td></tr><tr><td>Dev Support</td><td>0.123</td><td>0.040</td><td>0.306</td><td>3.064</td><td>0.004</td><td>0.042</td><td>0.204</td><td>1.165</td></tr><tr><td>User studies</td><td>0.138</td><td>0.047</td><td>0.286</td><td>2.921</td><td>0.006</td><td>0.043</td><td>0.233</td><td>1.116</td></tr><tr><td>UI Prototyping</td><td>0.133</td><td>0.057</td><td>0.253</td><td>2.346</td><td>0.023</td><td>0.019</td><td>0.247</td><td>1.354</td></tr></tbody></table>

Brace et al. suggest that the standardised coefficients of the predictor variables in a multiple regression can be used to compare the relative contribution of each predictor variable to the criterion variable and assess the strength of the relationship [24]. We could possibly assign weights to the HCI activities derived in proportion to these standardised coefficients as shown in column 5 of **Table 6**.

<table><caption>**Table 6:** A comparison of our recommended weights, average weights assigned by participants, weights derived by scaling up adjusted R² values from simple linear regressions (SLRs) and from scaling up the standardised coefficients of the stepwise multiple regression (MR).</caption><thead><tr><th rowspan="2">HCI Activity</th><th rowspan="2">Recommended weights</th><th rowspan="2">Assigned weights</th><th colspan="2">Derived weights scaled from</th></tr><tr><th>SLRs</th><th>MR</th></tr></thead><tbody><tr><td>1. User studies, user modelling...</td><td>3 – 4</td><td>3.7</td><td>3.1</td><td>4.3</td></tr><tr><td>2. Ideation with a multidisciplinary team</td><td>2</td><td>2.5</td><td>2.1</td><td>-</td></tr><tr><td>3. Product definition</td><td>1 – 3</td><td>3.1</td><td>2.4</td><td>-</td></tr><tr><td>4. Usability evaluation 1...</td><td>1 – 3</td><td>2.0</td><td>1.7</td><td>-</td></tr><tr><td>5. User interface prototyping</td><td>4 – 5</td><td>4.5</td><td>5.0</td><td>3.8</td></tr><tr><td>6. Usability evaluation 2 of the UI...</td><td>4 – 5</td><td>3.8</td><td>4.4</td><td>5.0</td></tr><tr><td>7. Development support...</td><td>3</td><td>3.2</td><td>4.4</td><td>4.6</td></tr><tr><td>8. Usability evaluation 3...</td><td>1 – 3</td><td>1.9</td><td>1.5</td><td>-</td></tr></tbody></table>

# 6 Conclusions

Drawing from literature, we proposed a framework comprising of 8 HCI activities. By using simple linear regressions, we could demonstrate that each of these activities had a significant positive correlation with the usability metric UGAM. In a stepwise multiple regression, four of these HCI activities accounted for 58% of the variation UGAM. We can conclude that while all activities in the framework affect usability,
the identified four HCI activities are relatively more important. The statistical analyses were in consonance with our original recommendations and with the weights assigned by practitioners, as summarised in **Table 6** above. Perhaps the most underestimated HCI activity during recommendation and assignment was the support that HCI teams need to give during the software development, though it was not a complete surprise.

A possible critique of our method could be that we showed the recommended weights to the participants before they assigned theirs. While this could have been an approach, it must be noted that that neither the recommended weights, nor the weights assigned by participants play a role in the regression analyses, which are based on the UGAM scores and activity scores alone. The weights derived from the regression analyses validate both the recommended and the assigned weights.

Another possible critique could be about our assumption that the scores of HCI activities are independent variables. Although the activity scores are naturally related (teams likely to score well on some HCI activities are likely to score well on others), it was essential to use them as predictor variables as it is the only way to establish their effect on usability. We minimised the bias by prescribing guidelines for evaluating each activity. The statistics did not show any multi-collinearity among the HCI activity scores.

Knowing which HCI activities are important would be useful in many contexts, particularly when resources are scarce and tradeoffs need to be made. Designers can use the rigorous, higher fidelity methods on activities that are more important, and make do with discount methods on less important activities. This knowledge would be useful in integrating HCI activities in software engineering processes – HCI professionals can insist on including the important activities, while conceding the relatively less important ones. The weighted average of the scores of activities could be used as a process metric as we describe in [18] and [19].

We used our framework of HCI activities, the waterfall model, UGAM as the product metric, and projects from the Indian IT industry to find the relative contribution of various HCI activities. Our results may be generalised within these choices. Other researchers could use other frameworks, other process models, other product metrics, and / or other contexts in a similar way to identify the activities that matter in those contexts.

## Acknowledgements

We thank Sanjay Tripathi, Pramod Khambete, Ved Prakash Nirbhay, Deepak Korpal, Atul Manohar, Aniruddha Puranik, Prof. UA Athavankar, Prof. Umesh Bellur, Prof. S Sudarshan, and all participants in the studies for their suggestions, assistance, and contributions.

## References

1. Archer, B.: Systematic Method for Designers. Council of Industrial Design (1965)

2. Jones, C.: Design Methods, Seeds of Human Futures. Wiley - Interscience (1970)
3. Lawson, B.: The Design Process Demystified. Butterworth Architecture (1980)

4. Cross, N.: Engineering Design Methods 3rd ed. John Wiley & Sons (2000)

5. Laseau, P.: Graphic Thinking for Architects and Designers. Van Nostrand Reinhold Company (1980)

6. Gould, J., Lewis, C.: Designing for Usability: Key Principles and What Designers Think. Communications of the ACM 28(3) (1985)

7. Nielsen, J.: Usability Engineering. Morgan Kaufmann (1993)

8. Kreitzberg, C.: Managing for Usability. In: Multimedia: A Management Perspective. Wadsworth (1996)

9. Dix, A., Finlay, J., Abowd, G., R., B.: Human-Computer Interaction, Second Edition. Prentice Hall (1998)

10. Preece, J., Rogers, Y., Sharp, H.: Interaction Design, Beyond Human-Computer Interaction, 1st edition. John Wiley & Sons (2002)

11. Sharp, H., Rogers, Y., Preece, J.: Interaction Design, Beyond Human-Computer Interaction, 2nd Edition. Wiley India (2007)

12. Beyer, H., Holtzblatt, K.: Contextual Design. Morgan Kaufmann (1998)

13. Cooper, A., Reimann, R.: About Face 2.0. Wiley (2003)

14. Mayhew, D.: Usability Engineering Lifecycle. Morgan Kaufmann (1999)

15. Garrett, J.: The Elements of User Experience. New Riders (2003)

16. Gulliksen, J., Göransson, B., Bovie, I., Blomkvist, S., Persson, J., Persson, J., Cajander, A.: Key Principles for User-centred System Design. Behaviour and Information Technology 22(6) (2003)

17. Buxton, B.: Sketching User Experiences. Morgan Kaufmann (2007)

18. Joshi, A., Tripathi, S.: User Experience Metric and Index of Integration: Measuring Impact of HCI Activities on User Experience. In: International Workshop on the Interplay between Usability Evaluation and Software Development, Pisa (2008)

19. Joshi, A., Sarda, N., Tripathi, S.: Measuring Effectiveness of HCI Integration in Software Development Processes. Journal of Software Systems In Press, Corrected Proof (2010) DOI: 10.1016/j.jss.2010.03.078.

20. Neill, C., Laplante, P.: Requirements Engineering: The State of the Practice. IEEE Software 20(6) (2003)

21. Joshi, A., Sarda, N.: HCI and SE: Towards a 'Truly' Unified Waterfall Process. In: HCI International, Beijing (2007)

22. Joshi, A.: Usability Goals Setting Tool. In: 4th Workshop on Software and Usability Engineering Cross-Pollination: Usability Evaluation of Advanced Interfaces, Uppsala (2009)

23. Joshi, A.: Index of Integration. (Accessed 2009) Available at: http://www.idc.iitb.ac.in/~anirudha/ioi.htm

24. Brace, N., Kemp, R., Snelgar, R.: SPSS for Psychologists, 2nd edition. Palgrave Macmillan (2003)
AFFINE for enforcing earlier consideration of NFRs
and human factors when building socio-technical systems
following agile methodologies

Mohamed Bourimi¹², Thomas Barth², Joerg M. Haake¹, Bernd Ueberschär³, Dogan Kesdogan³

¹FernUniversität in Hagen, Cooperative Systems, 58084 Hagen, Germany

²University of Siegen, Chair for IT Security, 57076 Siegen, Germany

³Leibniz Institute of Marine Sciences at the University of Kiel, 24105 Kiel, Germany

**Abstract.** Nowadays, various user-centered and participatory design methodologies with different degree of agility are followed when building sophisticated socio-technical systems. Even when applying these methods, non-functional requirements (NFRs) are often considered too late in the development process and tension that may arise between users' and developers' needs remains mostly neglected. Furthermore, there is a conceptual lack of guidance and support for efficiently fulfilling NFRs in terms of software architecture in general. This paper aims at introducing the AFFINE framework simultaneously addressing these needs with (1) conceptually considering NFRs early in the development process, (2) explicitly balancing end-users' with developers' needs, and (3) a reference architecture providing support for NFRs. Constitutive requirements for AFFINE were gathered based on experiences from various projects on designing and implementing groupware systems.

# 1 Introduction

Nowadays, a shift is taking place from single-user-centered usage to support multi-user needs and hence covering many collaboration measures and social aspects. The needed technical support for these users' activities in many important areas of our professional and leisure life activities is provided through collaborative applications also known as groupware as well as social software. According to Shneiderman and Plaisant “an extrapolation of current trends leads to the suggestion that most computer-based tasks will become collaborative because just as most work environments have social aspects” [1]. Thus, software systems and applications supporting collaboration are considered as socio-technical systems in the Computer-Supported Cooperative Work (CSCW) as well as Human-Computer Interaction (HCI) research fields [2]. Because socio-technical systems are characterized by complex scenarios which are mostly reflected e.g. in the user interface, HCI and CSCW also focus nowadays on human aspects of the development of computer technology in collaborative settings. While the goals of interaction are mostly covered by functional requirements (FRs), users' preferences (e.g. usability) and concerns (such as privacy
and security) are related to non-functional requirements (NFRs). According to [3],
FRs define what the system does and therefore its functionality whereas NFRs define
how a system has to be. Many CSCW and HCI key literature studied NFRs such as
usability in socio-technical and the trade-offs, which could arise between them, e.g.
privacy and awareness trade-offs in those systems. However, various literature state
that current approaches do not adequately consider generally NFRs from the
beginning in the development processes such stated in [3]. Thus, recently many
development approaches especially in the area of socio-technical systems follow user-
centered and participatory design in combination with agile methodologies in order to
efficiently react on end-users' emerging needs and (change) requirements [4,5,6]. In
our opinion, even when a given NFR is considered from the beginning (i.e. usability
in user-centered or participatory design), it is mostly contemplated separately from
other NFRs and factors. When considering that socio-technical systems mostly
represent a special category of distributed systems that are known to be difficult to
design and maintain, tensions could arise between project stakeholders (i.e. end-users
and developers) especially in agile settings. Furthermore, current approaches often not
explicitly address the gap of mapping NFRs into the underlying system architecture.
In this paper, we present the AFFINE (Agile Framework For Integrating Non-
functional requirements Engineering) simultaneously addressing these needs.

We first present identified needs in Section 2. Next, we describe our approach
consisting of the AFFINE framework in Section 3 and our conclusions in Section 4.

## 2 Problem and Requirements Analysis

Software development processes can be seen as complex collaborative social
processes. In order to reduce the potential complexity of these processes and assure
the delivery as well as the quality of the products, many models (e.g. the well-known
waterfall, prototyping, and spiral model) tried to structure the software development
processes and define their behavior e.g. by introducing roles and defining software
development life cycles. Latter include common phases like the requirements
analysis, design, development, testing, and support phases. In contrast to the classical
defined process models, agile process models and methodologies intend a better
reaction on unexpected problems often by consideration of human factors. They are
empirical processes that cannot be consistently repeated and therefore require
constant monitoring and adaptation [7]. However, Balzert states in [8] that according
to a coarse classification of the activities independently of a given development
processes, one could generally differentiate between two main phases, namely, the
solution specification phase and the solution construction phase. While most of the
activities of the specification phase can be classified as requirements engineering
activities, the activities of the construction phase target mapping a given solution
specification to a concrete technical solution. Different software engineering practices
recognized the critical importance of NFRs for the specification and construction of
software systems in general. A classical work addressing NFRs is [3] state that
software engineering practices concentrate on FRs, rather than NFRs. Furthermore,
the authors cite that NFRs are generally stated informally during the requirements
analysis, are often contradictory, difficult to enforce, and to validate during software
development process. Based on further literature, they state that not taking NFRs
properly into account is acknowledged to be the most expensive and difficult to
correct once the software has been implemented and thus, there is a need to deal
comprehensively with such requirements during the system development process. The
concrete needs we address in this paper were identified based on one of the long-
running project CURE (Collaborative Universal Remote Education) we were able to
follow. This project has a very representative character since its needs correspond to
identified needs in other literature. The CURE platform was developed at the
FernUniversität in Hagen (FuH) to support different collaborative learning and
collaborative work scenarios [9]. The development process followed in CURE is an
agile process called the Oregon Software Development Process (OSDP) described in
[10]. Applying OSDP considered end-users' feedback of the participating
departments at the FuH. Representatives of students and instructors from various
disciplines such as mathematics, electrical engineering, computer sciences and
psychology were participating in the usage and evaluation of the prototypes resulting
from each OSDP-iteration. Even though OSDP considers conceptually NFRs in form
of a NFR backlog, their consideration was not earlier enough to overcome drawbacks
in the construction phase. In the case of CURE, responding to end-users wishes
related to NFRs (e.g. usability of the web interface, performance of the synchronous
communication means and awareness provision in the shared workspaces) was
interrupted in order to meet the delivery, integration deadlines and budget. CURE was
extended in various sub-projects (e.g. [11,12,13]). Most of these works were primarily
concerned with improving NFRs which were classified as insufficiently covered by
the developed system or tried to address new needs emerged through the usage of the
system. Ambler states in [14] that NFRs and constrains are difficult to consider in
projects following agile methodologies. A conceptual consideration of NFRs in the
followed methodology avoids delegating their fulfillment to the intuition of involved
people that could result in intentional or accidental negligence. Thus, we identify the
need of *conceptually enforcing the consideration of all relevant NFRs and possible
trade-offs early in the development process (N1)*.

Involving end-users in an agile process could be very expensive. Especially when an
agile methodology is followed in the end-users as well as developers are often
experiencing continuous communication tensions. Developers are often asked to
change, e.g., user interfaces or functionality, which seemed to be agreed upon earlier.
Furthermore, on the one hand end-users and developers have different terminology
for the same things or the same terminology for different things. On the other hand,
members of the same development team might have different backgrounds and
terminologies. This is also crucial in the case that different partners and/or distributed
teams are cooperating in the same project. Communication problems are well known
in the software engineering field and do not concern only agile methods. The same
methodology may not be imposed to different stakeholder in the project, since
involved parties may already have elaborated methodologies and processes as well as
have different interests and goals (i.e. using their own software pieces or products
etc.). Indeed, recent studies show that the most frequent failure source are
communication problems with more than 70% [15] and that 33% of the projects are
negatively affected or cancelled because of changing the requirements [16]. Based on
our experiences we argue that this is especially expensive when following an agile
development process. Even though agility assure the close involvement of end-users,
latter are mostly not experienced in communicating requirements to the developers
[8]. Thus we identify the second need of explicitly balancing end-users' with
developers' needs when following agile development method(ologie)s (N2).

The design and evaluation of socio-technical systems is still a challenge because of
the exploratory nature of these applications [1]. Indeed, people involvement varies
and the usage can range from occasional to frequent according to a given setting and
circumstances. The same socio-technical system can lead to different evaluation
results in different social environments [2]. The evaluation of socio-technical systems
needs methodologies and approaches that allows for rapid and cost-effective
development and usage of prototypes. Shneiderman and Plaisant mention that “while
software engineering methodologies are effective in facilitating the software
development process, they have not always provided processes for studying the users,
understanding their needs, and creating a usable interface” [1]. Depending on the
project specific situation, development costs need often to be reduced. Software
should not be built from scratch each time in the development process. In [17],
Grudin addresses challenges for groupware developers and suggests that adding new
functionality to an accepted application is more adequate than developing a new
application. This is a typical case when building many socio-technical systems or
while their evolution. At a first glance, adding new functionality and enhanced
interaction possibilities to existing systems seems attractive. However, adding new
functionality often requires adding and modifying a lot of source code. This often
complicates the API and requires a redesign of the domain model. Furthermore, Paech
et al. argues in their position paper [18] that FRs and NFRs as well as architecture
should not be separated. The emerging changes are especially crucial when
considering costs in terms of (re-)design, implementation and retrofitting costs. Thus,
added functionality to socio-technical systems is reflected in growing complexity of
their classes and/or components. Thus, the extension or retrofitting and the integration
of new components in these systems represent realistic scenarios, which have to be
considered in terms of development costs. However, it is important that by freezing
changes, the design of the system stays extendable for future extensions and
retrofitting. Thus, we formulate the third need as follows: *The development method
must be supported at the architectural and construction level to assure meeting N1
and N2 at minimal cost. A Kind of reference architecture providing support for
NFRs is needed (N3).* While N1 is more concerned with the specification phase of a
given socio-technical system and N3 with its construction phase, N2 still overlapping
both phases when following an agile methodology. *Simultaneous* consideration of
N1-N3 is therefore required.

3 Our Approach: AFFINE

Introducing an agile method at the level of the development process is the key to
satisfy N1 and N2 at the organizational level in our opinion. In order to reduce the
complexity of the involvement of our method in various phases of the followed
development process in a given project, we propose as an integral component Scrum [28]. Scrum can be seen as a process for empirical control of software development, which helps in handling changing requirements more efficiently by considering human factors in the development process of both; customers (in our case end-users) and project stakeholders in general. Based on our experiences in various projects, the main strengths of Scrum consists of (1) the simplicity in terms of roles defined (Scrum master, product owner, and development team), development steps to be followed (e.g. development periods called sprints), documentation to be produced (e.g. sprint backlog), and meetings to be held (e.g. daily Scrum), (2) balancing the needs of the customers and developers through consensus enforcement for a given deliverable and continuous communication (e.g. in the daily Scrum meetings), (3) creating awareness on ongoing project tasks (also in daily Scrum e.g.), and (4) allowing for better as well as faster handling of detected, non-expected problems during the development process, which generally results (by right application) in better acceptance of the delivered product with low costs.

<figure><img src="image_81.png" /><figcaption>**Figure 3.1: The Scrum based AFFINE method.**</figcaption></figure>

The right side of Figure 3.1 represents broadly a typical Scrum development procedure. A facilitator (in our case the Scrum master) as well as the product owner and development team interact with each other in order to drive the product development. This interaction is represented as loop involving them all together. Since the facilitator moderates the interaction, we represent such loop as an arrow starting and ending in the facilitator role. In Scrum, a sprint backlog document might be updated in such loop. Numerating this loop in our representation does not imply that the interaction is carried out in a giving sequence inside it. The left side of the same Figure shows our extension of a typical Scrum process to enforce the earlier consideration of NFRs. There, we introduce the role of a NFR stakeholder (mostly experts), who is concerned with the fulfillment and consideration of a respective
NFR. The same facilitator has to moderate the circulation of the common document to the first stakeholder (according to prior prioritization), who has to update the document (e.g. adding warnings, requirements or changing them etc.). The structure of the common document and its content have to be defined from the project involved parties. However, circulating only one document, which contains all needed information for the development of given product, should avoid potential inconsistencies and information lost. After updating the document (at least by annotating that it was reviewed and maybe admitted without changes from the respective stakeholder), the document returns to the facilitator(s) and loops again over the left side. By admission without changes, the facilitator might shorten this iteration and directly forward the document to the next stakeholder. If any change happens in late circulations at the level of a stakeholder, the document returns to the facilitator, has to be circulated in the right side, and finally has to begin the circulation at the first stakeholder at the left side. We suggest the following informal steps for N1 and N2:

1. Involvement of all stakeholders of the project and introducing the role of the facilitator (one or more according to the project setting).

2. Goals or use cases (UC) identification of the intended processes (by defining the set of FRs). The facilitator has e.g. to guarantee the same terminology is used and has to detect miss-satisfaction signs in the different phases.

3. Alignment of all NFRs that have to be considered prioritizing them according to the project goals or UCs.

4. Responsible and experts for each goal or UC as well as NFR have to be chosen.

5. Circulating a single document (to avoid syncing various documents) containing the set of goals or UCs and their specification and modeling (by considering aligned FRs and NFRs at the same time). For this, UML or similar notations could be helpful in order to estimate efforts. The circulation has to be performed by the facilitator according to the priority of the NFRs. If a breakdown is identified in the circulation loop, the document has to be send back to the responsible of the first affected NFR (ordinarily with higher priority). If many NFRs are simultaneously affected, a meeting of the responsible and experts has to be organized. When conflicts arise, the facilitator intervenes in order to reach consensus. Since the facilitator is normally only a supporting role, his main goal consists in delivering the result while preserving satisfaction of end-users and project stakeholders. However, the final decision has to be made by the responsible(s) or at least by the coordination entity of the project.

6. The circulation ends when reaching the goals i.e. by implementing the UCs and testing them (also through the end-users).

Those steps have to be executed for each project iteration. If the project is organized according big work packages, following a divide and conquer methodology could be helpful. In order to optimize the requirements gathering, user-centered design and modeling steps (for instance by using established methods like prototyping) and UML or ER diagrams (as mentioned before) could be useful.

Finally, we want to mention, that the facilitator does not represents a critical point in this procedure. Any person familiarized with development activities should be able to act within this process as a facilitator. Further, if Scrum is integrated as an agile method, the Scrum certification exam ensures needed qualification of a facilitator.
Related to N3, SOA is currently assessed as the next step forward in the design,
development, operation, and organization of large-scale distributed systems (see e.g. [22]). Characteristics like loose coupling, discovery of artifacts during design/run time, and the ability to reuse services to enable efficient adaptation of a system to changing requirements (e.g. changes in users behavior, processes) are not supposed to be provided by an architectural approach for the first time. Learning from preceding approaches, characteristics like the commitment to open standards and the separation of architectural concepts from their technical implementation led to a widespread acceptance of service-oriented principles in commercial as well as scientific communities. Since in our context NFRs are in the focus of attention, the inherent possibility of tailoring an SOA at design time according to the actual needs by carefully selecting the specifications to implement is an adequate means to realize these requirements [22]. Beyond this, NFRs like usability and performance in socio-technical systems can only be assessed during runtime and in close cooperation with the end-user. As already discussed in this paper, an early consideration of NFRs during the lifecycle of an SOA leads to reduced development cost. Integrating different stakeholders into the development of an SOA is one approach to handle this; a Service Life Cycle Model focusing on SOAs stakeholders as a prerequisite for governing an SOA throughout its lifecycle is presented in [23]. Since NFRs are crosscutting concerns, the positive synergy between aspect-oriented programming (AOP) techniques and SOA for satisfying NFRs implementation was beneficial in our case. The architecture of the CURE-based sub-projects described in the related publications supports different kind of clients (i.e. Eclipse RPC thick client for the collaborative design editor [11], normal and AJAX browser clients for the retrofitted CURE [12], and mobile as well as ubiquitous clients for the ubiquitous CURE [13]) with the same SOA/AOP layer. Thereby, different kinds of architecture families also are supported (based on the on client-server model, replicating or P2P). Surely, this is due because the context of these sub-projects could be satisfied with such single layer. Nevertheless, if different contexts have to be supported, various instances of the SOA/AOP layer could be deployed. So we mean that our AOP/SOA-based generic architecture meets N3 with a high genericity.

# 4 Conclusion

In this paper, we proposed the AFFINE framework that aims at simultaneously addressing three needs when developing socio-technical systems by following agile methodologies. The three needs were identified from the long-running project CURE as well as based on relevant HCI and CSCW literature gathered experiences. The main idea of AFFINE is to use an agile method including Scrum as an integral part. The agile method is user-centered and considers human factors, which could affect the success and acceptance of the developed socio-technical systems. The method enforces conceptually the earlier consideration of NFRs while the suggested supporting architecture provides a generic reference architecture for developing socio-technical systems in agile development settings. Furthermore, AFFINE is independent from a specific software development process and applicable for different phases of the followed process in a given project. The concrete suggestion to use SOA and AOP
at the technological level showed their advantages in first evaluations. The proposed
framework is now successfully being applied in the ongoing work of many projects
led by us as first empirical evaluations show. Future work aims at collecting more
experiences and refining AFFINE.

5 References

1. Shneiderman, B., Plaisant, C.: Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th Edition). Pearson Addison Wesley (2005).

2. Gross, T., Koch, M.: Computer-Supported Cooperative Workspace. Oldenburg (2007).

3. Chung, L., Nixon, B.A.: Dealing with non-functional requirements: three experimental studies of a process-oriented approach. In: ICSE '95, ACM (1995).

4. Jokela, T.: Assessment of user-centred design processes - lessons learnt and conclusions. In: PROFES. (2002).

5. Schümmer, T., Lukosch, S., Slagter, R.: Empowering end-users: A pattern-centered groupware development process. CRIWG 2005. Vol. 3706, Springer (2005).

6. Lieberman, H., Paterno, F., Wulf, V., eds.: End User Development. Springer (2006)

7. Highsmith, J.: Agile software development ecosystems. Addison-Wesley (2002).

8. Balzert, H.: Lehrbuch der Softwaretechnik, Basiskonzepte und Requirements Engineering, 3ed., Spektrum (2008).

9. Haake, J.M., Schümmer, T., Haake, A., Bourimi, M., Landgraf, B.: Supporting flexible collaborative distance learning in the cure platform. Vol. 1, IEEE Computer Society (2004).

10. Schümmer, T., Slagter, R.: The oregon software development process. (2004).

11. Bourimi, M.: Collaborative design and tailoring of Web based learning environments in CURE. In CRIWG 2006. LNCS 4154, Springer (2006).

12. Bourimi, M., Lukosch, S., Kühnel, F.: Leveraging visual tailoring and synchronous awareness in Web-based collaborative systems. In CRIWG. Volume 4715, Springer (2007).

13. Bourimi, M., Kühnel, F., Haake, J., Abou-Tair, D., Kesdogan, D.: Tailoring collaboration according privacy needs in real-identity collaborative systems. In: CRIWG. (2009).

14. Ambler, S.W.: Beyond functional requirements on agile projects. Dr. Dobb's Journal 33(10) (2008).

15. Huth, S.: Marktstudie probleme und fehler im requirements-engineering, www.sigs-datacom.de/wissen/artikel-fachzeitschriftel/artikelansicht.html?tx_mwjournals_pi1%5BshowUid%5D=2479 (Accessed 2010).

16. Emam, K.E., Koru, A.G.: A replicated survey of it software project failures. IEEE Softw. 25(5) (2008).

17. Grudin, J.: Groupware and social dynamics: eight challenges for developers. Communications of the ACM 37(1) (1994).

18. Paech, B., Dutoit, A.H., Kerkow, D., Knethen, A.V.: Functional requirements, non-functional requirements, and architecture should not be separated. Technical report, Proceedings of the International Workshop on Requirements Engineering (2002).

19. Schwaber, K.: Scrum overview. http://codebetter.com/blogs/darrell.norton/pages/50339.aspx (Accessed August 2010).

20. Erl, T.: Service-Oriented Architecture (SOA): Concepts, Technology, and Design. Prentice Hall PTR (2006).

21. Gu, Q., Lago, P.: A stakeholder-driven service life cycle model for SOA. In: IW-SOSWE '07: 2nd international workshop on Service oriented software engineering, ACM (2007).
# Understanding Formal Description of Pitch-Based Input

Ondřej Poláček and Zdeněk Míkovec

Faculty of Electrical Engineering, Czech Technical University in Prague, Karlovo
nám. 13, 12135 Prague 2, Czech Republic
{polacond,xmikovec}@fel.cvut.cz

**Abstract.** The pitch-based input (humming, whistling, singing) in acoustic modality has already been studied in several projects. There is also a formal description of the pitch-based input which can be used by designers to define user control of an application. However, as we discuss in this paper, the formal description can contain semantic errors. The aim of this paper is to validate the formal description with designers. We present a tool that is capable of visualizing vocal commands and detecting semantic errors automatically. We have conducted a user study that brings preliminary results on comprehension of the formal description by designers and ability to identify and remove syntactic errors.

**Keywords:** Non-verbal Vocal Interaction; Vocal Gesture; Formal Description; User Study

## 1 Introduction

The *Non-Verbal Vocal Interaction* (NVVI) can be described as a method of interaction, in which sounds, other than speech, are produced. There are several approaches described in the literature which include using pitch of a tone, length of a tone, volume, or vowels in order to control the user interfaces. The NVVI is an interaction method that has already received a significant focus within the research community. It has been used as an input modality for people with motor disabilities [7][3] as well as voice training tool [2]. It is a method that shares some similarities with *Automatic Speech Recognition* (ASR). However, when comparing both interaction styles, several differences are revealed. Several reports, including mouse emulation [1] or controlling real-time games [7], suggest that NVVI is better fitted to continuous control rather than ASR. NVVI is cross-cultural and language independent [8]. Unlike ASR, NVVI generally employs simple signal processing methods [3]. Due to NVVIs limited expressive capabilities, ASR is better at triggering commands, macros or shortcuts. NVVI should be considered as a complement to ASR rather than replacement.

To design an application controlled by speech a set of word patterns or grammar must be defined. This grammar will then allow the ASR to recognize a range of expected words used in utterances. Likewise, a designer can also use a similar formal method for pitch-based NVVI.
<figure><img src="image_82.png" /><figcaption>Fig. 1. NVVI signal processing pipeline</figcaption></figure>

The signal processing pipeline for most pitch-based NVVI systems is depicted in Figure 1. Pitch is extracted from the sampled signal in a short discrete periods of time called frames. The typical duration of one frame is approximately 20 ms. The formal description of the NVVI and a stream of frames are then matched together, followed by generation of an appropriate action.

## 2 Formal description

When designing a set of voice gestures, the designer must describe an ideal pitch profile for each gesture. These ideal pitch profiles are then referred to as *gesture templates* and they are usually represented in graphic form as shown in Figure 2. However, the users are unable to produce an ideal pitch profile. The interpretation of gesture templates by the user is referred to as *gesture instances*. An example of the relationship between a gesture template and its instances is depicted in Figure 2. Note that slightly different instances share the same semantics defined by the gesture template which is in this case an increasing tone. Once gesture templates are designed in a graphic form, they

<figure><img src="image_83.png" /><figcaption>Fig. 2. Relationship between a gesture template and its instances</figcaption></figure>

can be described by a *Voice Gesture Template* (VGT) expressions. Design of VGT expression is described in detail in [5]. These expressions are similar to regular expressions. They have two terminal symbols *p* and *s* that correspond to pitch and silence. They also use an operator *\** for repetition and operator | for the choice. However, there are several symbols with different meanings, for example brackets [*] which are used for more sophisticated conditions and brackets <> which are used for output definitions to trigger an action. The use of VGT expressions is illustrated in Figure 3. The gesture template depicted in Figure 3 describes instances which start under midi note 60 and increase in pitch to more than 4 midi notes. Midi notes [4] are numerical representations of
<figure><img src="image_84.png" /><figcaption>Fig. 3. VGT expression and its graphical representation of gesture template.</figcaption></figure>

traditional notes in western music notation, for example, midi note number 60 corresponds to c'. The Figure 3 also illustrates the relation of VGT expression and the graphic representation of the gesture template. This process can be divided into four parts:

1. In the first part, the frame p1 is matched to the expression when its pitch is under midi note 60. This is ensured by the condition [p1.m < 60] where the attribute .m is a midi note value of the frame p1;

2. Then all pitch frames p* are matched until the difference between the pitch of a current frame and the frame p1 is higher than or equal to 4 midi notes (frame p2). This is ensured by the condition [p2.m - p1.m >= 4];

3. After satisfying the condition in the 2nd step, all pitch frames p <move>* are matched and the output symbol move is triggered with each frame;

4. The processing of the template is completed, when a silent frame s is matched.

## 3 Semantic Errors

Semantic information, that describes pitch profiles of gesture templates, is encoded by a VGT expression. However, the description of gesture templates may be affected by semantic errors which cannot be detected while parsing the expression. A semantic error can also appear in a VGT expression when a new gesture template is added to the expression. The expression must be checked by tedious experimenting that involves user input to see if all templates are recognized correctly. Our research has identified two frequent types of semantic errors which cause improper behavior in gesture recognition – *ambiguous* and *unreachable* templates.

Two gesture templates are ambiguous if there is at least one gesture instance that satisfies both templates. The reason this error frequently occurs is due to an imprecise template description. In a real application there is typically a large number of instances fulfilling the condition of ambiguity. This semantic error is typically demonstrated by the generation of two or more output symbols in one frame.

The gesture template is unreachable when there is no instance matching the template. This can, for example, be caused by a condition that is always false, the template does not take into account human capabilities, or there is another gesture template that prevents the unreachable template from matching instances.
## 3.1 Semantic Error Detection

Detection of semantic errors, which are described above, requires analysis of gesture instances that can be generated by a VGT expression. We have implemented a tool which is capable of displaying possible gesture instances and automatically identifying semantic errors. It also allows deeper understanding of matching an instance to an expression by tracking its pitch profile. After generating all possible instances that match the expression, the tool checks if each instance belongs to just one template (ambiguity condition) and if each template has at least one instance (unreachability condition).

<figure><img src="image_85.png" /><figcaption>Fig. 4. Tool for vocal gestures visualization and semantic error detection.</figcaption></figure>

The user interface of our tool is depicted in Figure 4. Part A of the Figure shows a dialog which contains a list of templates and number of instances. The dialog shows semantic errors within the VGT expression by displaying both ambiguous and unreachable templates (see the *Status* column). The user can display instances by selecting an appropriate row. When selecting a row with ambiguous templates, instances cause the ambiguity to be displayed in part B.

Gesture instances are shown in the part B of Figure 4. The horizontal axis represents frames converted into timestamps in milliseconds and the vertical axis represents pitch using midi note numbers [4] starting with silence at the bottom. The black lines represent the generated gestures. When there are a lot of instances and their typical pitch profile is not visible, the user can display these instances and track them from the beginning to the end. When tracking an instance, the corresponding position of a VGT expression is highlighted in the VGT Expression Debugger (dialog in part C). Horizontal and vertical bars represent the current position, the bold line represents the part of an instance that has been already tracked and the blue lines show the further extending of a current instance.
The VGT expression is shown in dialog C. The current position of tracked instance is highlighted directly in the VGT expression by a yellow background, allowing the user to inspect how the instance is matched to its template. This is a very useful feature when inspecting instances that correspond to two or more ambiguous gestures, as the user can now clearly see the cause of the ambiguity. Current pitch values of numbered pitch frames are shown below the expression.

## 4 User study

The aim of the user study was to find out whether the designers could understand VGT expressions, and to demonstrate the usefulness of the tool described in the previous section. Eight designers were recruited to participate in the study. Each participant (mean age=29.6, SD=2.8) had some previous experience with NVVI – four of them knew the interaction method, three had used it at least once and one had previously designed an NVVI application. Seven of the participants considered themselves as interaction designers and the remaining one as a usability expert. All participant were familiar with regular expressions.

The participants were given approximately 20 minutes of training, which involved discussing the syntax of two VGT expression examples as well as semantic errors. The participants were asked to complete three tasks. In each task they were told to recognize the gesture templates in given VGT expression by describing them orally and sketching a graphic representation of each template. They were also asked to identify any semantic errors that may have been present in the expressions and to propose a solution for each. However, they were not told to write a new corrected expression due to limited time of each session. One session lasted approximately one hour. Participants were divided into two groups of four – Group A and B. *Group A* was allowed to use the tool described above, whereas *Group B* was not allowed to use any aid.

### Task #1

In the first task participants were told to analyze the following VGT expression:

```
p1 p* (p2 [p2.m - p1.m > 4] p* s <alpha> |
       p3 [p2.m - p2.m > 8] p* s <bravo>)
```

The expression above describes the two templates as depicted in Figure 5a. The *alpha* template defines instances where pitch increases by 4 or more midi notes. The *bravo*'s instances have to increase by 8 midi notes. However, the *bravo* template is unreachable, as the condition in the *alpha* template is always matched earlier.

**Group A** (Use of tool): Each participant correctly understood the templates and discovered that the gesture *bravo* was unreachable. Two participants proposed a partially correct solution.

**Group B:** One participant misunderstood the *bravo* template and consequently could not see an error. The other participants miscategorized the error as ambiguous. Two participants proposed a partially correct solution.
<figure><img src="image_86.png" /><figcaption>Fig. 5. a. Gesture templates in the task #1 b. Gesture templates in the task #2</figcaption></figure>

## Task #2

The second task contained gestures used in the *Tetris* game controlled by humming [7]. The participants were again told to analyze the VGT expression:

```
p1 p*
(p2 [p2.m - p1.m > 4] p <alpha>* s |
p3 [p1.m - p3.m > 4] p <bravo>* s) |
p*200;600 s <charlie> |
p*500; s <delta>
```

The expression above describes the templates depicted in Figure 5b. *Alpha* instances have to increase in pitch by 4 or more midi notes, whereas the *bravo* instances have to decrease by the same amount. *Charlie* instances are short tones of 200 to 600 ms and *delta* instances are all those that are longer than 500 ms. Two ambiguities are present in the expression. The first one is a time overlap in *charlie* and *delta* templates. The solution is to modify one of the limits. The second error is a pitch overlap between *alpha*, *bravo* and *charlie*, *delta* templates, due to the latter two not defining a pitch limit. The solution is to limit the pitch in *charlie*, *delta* templates to within ±4 midi notes.

**Group A (Use of tool):** Each participant understood the presented templates. One participant incorrectly identified the gestures initially, but corrected their interpretation after using the tool. All four were also able to locate all errors and propose a correct solution for each error.

**Group B:** Unlike the three others, one participant was not able to describe *alpha* and *bravo* templates correctly. All four participants were able to find ambiguity between *charlie* and *delta*. The second error was found by three participants, who proposed a correct solutions for each of the errors.

## Task #3

The most complex VGT expression was analyzed in the last task. The expression defines six of the eight templates used in keyboard controlled by humming [6].

```
p11 [p11.m< 60] p12 [p12.m-p11.m<=4 & p11.m-p12.m<=4]* s<alpha> |
p21 [p21.m>=60] p22 [p22.m-p21.m<=4 & p21.m-p22.m<=4]* s<bravo> |
p31 [p31.m< 60] p* p32 [p32.m-p31.m>4] p* s<charlie> |
p41 [p41.m>=60] p* p42 [p41.m-p42.m>4] p* s<delta> |
p51 [p51.m< 60] p* p52 [p52.m-p51.m>4] p* p53 [p53.m<=p51.m] p* s<echo> |
p61 [p61.m>=60] p* p62 [p61.m-p62.m>4] p* p63 [p63.m>=p61.m] p* s<foxtrot>
```
The six instances correspond to the following - 1. *alpha* to a straight low tone, 2. *bravo* to a straight high tone, 3. *charlie* to increasing tone by more than 4 midi notes, 4. *delta* to decreasing tone by more than 4 midi notes, 5. *echo* to a tone that increases by more than 4 midi notes and then decreases to at least its initial pitch and finally 6. *foxtrot* which is essentially *echo* vertically inverted. Ambiguities between *charlie* and *echo* and between *delta* and *foxtrot* are present due to the end pitch of *charlie* and *delta* templates not being limited.

<figure><img src="image_87.png" /><figcaption>Fig. 6. Gesture templates in the task #3</figcaption></figure>

**Group A (Use of tool):** One participant misunderstood *alpha* and *bravo* templates. Two participants incorrectly identified the templates initially, but corrected their interpretations after using the tool. Two participants thought there was an error present between *alpha* and *bravo*, but identified their mistake after using the tool. All participants located the error and three of them were able to propose correct removal solution.

**Group B:** Two participants incorrectly identified *alpha* and *bravo* templates as unreachable and were thus unable to sketch them. The other two participant incorrectly identified the templates as ambiguous. However, the other templates were understood by all participants, who were also able to identify the ambiguities and propose correct solutions.

# 5 Discussion

Using VGT expressions accelerates the process of building an NVVI application, as the matching algorithm no longer needs to be hard coded. The question, that is raised though, is whether designers are able to understand these VGT expressions. In most cases, participants from both groups correctly identified templates directly from VGT expression, which supported our assumption that VGT expressions can be understood by most designers. From total of 48 gestures that were examined in one group, there was two errors in the group A (use of tool) and seven error in the group B. What was slightly surprising was that participants from group A primarily relied on their own judgement rather than on the provided tool. However, they did use the tool from time to time to visually confirm their opinion or when they were unsure of the answer. In these situations the tool helped them to correctly understand the given templates and consequently to succeed in fulfilling the tasks. Thanks to the tool, participants from the group A also had no difficulty in detecting semantic errors. Although
the participants from the group B were not as successful as group A, they were still able to locate a significant number of error occurrences. It seems that the use of the tool results in better understanding of VGT expressions and minimizes the overlooking of semantic errors. However, a further quantitative study is needed in order to support this hypothesis.

## 6 Conclusion

This paper discusses the formal description of pitch-based vocal input, used during the design process of NVVI applications. We have created a tool for automatic error detection and visualization of the formal description. Our research was focused on the comprehension of the formal description by designers and their ability to detect possible semantic errors with and without using the tool. Their ability to comprehend the formal description and to detect semantic errors was validated in a user study by eight interaction designers. Designers who used the tool were more successful in understanding the formal description. Further research concerning these results will be conducted in the future, including a comparative quantitative study to prove the efficiency of the gesture visualization tool.

**Acknowledgments.** This research has been partially supported by the MSMT research program MSM 6840770014 and the VitalMind project (IST-215387).

## References

1. Harada, S., Landay, J.A., Malkin, J., Li, X., Bilmes, J.A.: The vocal joystick: evaluation of voice-based cursor control techniques. In: Proceedings of ASSETS'06, pp 197-204. ACM Press (2006)

2. Hämäläinen, P., Mäki-Patola, T., Pulkki, V., Airas, M.: Musical computer games played by singing. In: 7th International Conference on Digital Audio Effects, pp 367-371. (2004)

3. Igarashi, T., Hughes, J.: Voice as sound: using non-verbal voice input for interactive control. In: Proceedings of UIST'01, pp. 155-156. ACM Press (2001)

4. MIDI Manufacturers Association: Complete MIDI 1.0 Detailed Specification v96.1 (2nd ed.), http://www.org/techspecsmidipec.php (2001)

5. Poláček, O., Míkovec, Z., Sporka, A.J., Slavík, P.: New way of vocal interface design: Formal description of non-verbal vocal gestures. In: Proceedings of the CWUAAT'10, pp. 137-144. Cambridge Press, UK (2010)

6. Sporka, A.J., Kurniawan, M., Slavík, P.: Non-speech Operated Emulation of Keyboard. II: Designing Accessible Technology, ISBN 1-84628-364-7, pp. 145-154. Springer, Heidelberg (2006)

7. Sporka, A.J., Kurniawan, S.H., Mahmud, M., Slavík, P.: Non-speech Input vs Speech Recognition: Real-time Control of Computer Games. In: Proceedings of ASSETS'06, pp. 213-220. ACM Press (2006)

8. Sporka, A.J., Žikovský, P., Slavík, P.: Explicative Document Reading Controlled by Non-speech Audio Gestures. In Text, Speech and Dialogue, 9th International Conference, TSD 2006, LNAI 4188, pp. 695-702. Springer, Heidelberg (2006)
Application Composition Driven By UI Composition

Christian Brel, Philippe Renevier-Gonin, Audrey Occello, Anne-Marie Déry-Pinna,
Catherine Faron-Zucker and Michel Riveill,

I3S Laboratory (UMR 6070 - Université Nice - Sophia Antipolis et CNRS)
930 routes des colles - BP 145

{brel, renevier, occello, pinna, faron, riveill}@polytech.unice.fr

**Abstract.** Ahead of the multiplication of specialized applications, needs for application composition increase. Each application can be described by a pair of a visible part –the User Interface (UI) –and a hidden part –the tasks and the Functional Core (FC). Few works address the problem of application composition by handling both visible and hidden parts at the same time. Our proposition described in this paper is to start from the visible parts of applications, their UIs, to build a new application while using information coming from UIs as well as from tasks. We base upon the semantic description of UIs to help the developer merge parts of former applications. We argue that this approach driven by the composition of UIs helps the user during the composition process and ensures the preservation of a usable UI for the resulting application.

**Keywords:** User Interface Composition, Application Composition,

# 1 Introduction

There are more and more software tools: on the web, on Smartphones, on laptops, etc. Having so many widgets is interesting; however, to reach a friendly use, there is a need to compose them. For example, a Smartphone can provide its user with a diet list and an application that gives restaurants close to her and their menus. She would probably enjoy a second application filtering or emphasizing dishes from her diet list.

To construct new applications by reusing other application sub-parts is a key challenge of Software Engineering. This is a mean to speed up development cycles. An interactive systems is composed at least of a functional part, usually called Fonctional Core (FC),and a User Interface (UI). Moreover, in the HCI research field, there is a strong recommendation of using a Task Model (TM) during requirements analysis. The TM describes the needs and the procedures to achieve these needs. The TM is not often explicitly implemented, but it can express the relationship between FC and UI entities. We choose to use UI as primary artifacts of the composition process because UI are the parts of applications manipulated by both developers and ergonomic designers. We aim at enabling them to reuse existing UI for creating new applications while preserving user requirements of individual original systems and keeping some of the links between the FC part and the UI part in the resulting system.
In this paper, we propose to combine information at the three levels: FC, UI and TM. For this, we base the composition process on the selection, extraction and placement of the existing application's UI as elementary composition actions to impact underlying task trees and FC part. The remainder of this paper is organized in 5 sections, respectively, the description of other UI composition works, the presentation of our model used in our composition, the overview of our composition process, the description of our implementation of the global process and finally the conclusion.

## 2 Related Work

This section presents related work on UI composition grouped by their entry point in the composition process according to the application cutting: the Functional Core (FC), the Task Model (TM) and the User Interface (UI). Each entry point addresses a specific problem of composition: presentation and layout consideration at the UI level, behavior of the application at the FC level, user needs at the TM level. We classify works related to UI compositions according to their approach: an "X" in the Table 1 means that corresponding work explicitly takes into account this part.

<table><caption>**Table 1.** Classification of composition approaches.</caption><thead><tr><th></th><th>FC</th><th>UI</th><th>Tasks</th></tr></thead><tbody><tr><td>Developing adaptable user interfaces [9]</td><td></td><td>X</td><td></td></tr><tr><td>Amusing [8], ComposiXML [3]</td><td></td><td>X</td><td></td></tr><tr><td>C3W [11]</td><td></td><td>X</td><td></td></tr><tr><td>Task Models Merging [4]</td><td></td><td></td><td>X</td></tr><tr><td>Servface [7]</td><td></td><td>X</td><td>X</td></tr><tr><td>Compose [2]</td><td></td><td>X</td><td>X</td></tr><tr><td>Scenarios [12]</td><td></td><td>X</td><td>X</td></tr><tr><td>SOAUI [9], ALIAS [6], Transparent Interface[13]</td><td></td><td>X</td><td>X</td></tr></tbody></table>

We group related works in four categories:

* Works only considering UI composition, either for defining specific toolkit for adaptive UI [9], either based on abstract definition of UI [8,3] or either adopting end-user programming [11],

* Works only considering TM composition (composition of two task trees [4]),

* Works deriving Tasks in FC composition and later in UI composition, because of generation UI from service annotation [7] or thanks to specific adaptable couple FC-UI [2] or deriving Tasks in UI [12]

* And Works considering both FC and UI composition. The main goal in [13] is to maintain a stable UI for using a composition of volatile service. The SOAUI approach [9] derives web service composition into UI composition, by searching the best-fitting UI in a repository for each service and then UI composition. The aim of [6] is to deduce the UI composition from the FC composition.
We notice a lack in underlying composition processes. Either the design of original applications' UI with man-crafted properties such as ergonomic or usability is lost, or both FC and UI parts are no longer connected together in the resulting application, or there is no UI reuse. In the context of fast development processes, reusing UI without keeping ergonomic and usability criteria is useless. Loosing links between the UI and the FC parts engenders human interventions to connect the two parts which is error prone and fastidious for large applications.

So we propose in our approach to mix information from all the levels to improve the application composition. The collaboration between the three levels are expressed in a unifying model, we call Enhanced Task Tree (ETT), presented in the next section.

# 3. Enhanced Task Tree

In our approach, the process is guided by the composition of former UI and by their reuse to build the new application. Our work is based on a model that lets consider information from Functional Core (FC) and UI and from Task Model (TM).

## 3.1 Connecting Conception and Implementation of the Interactive System

We assume the decomposition in two parts of an application: the FC and the UI. Links between both parts are difficult to analyze in the code, so we use an external description with references to some running objects. We use the task model (TM) as a pivot. The TM is established at design time from requirements and user models. We enhance it with information from the running objects. For each initial application and for each task, we add semantic annotations. In the following, we call "UI block" one piece or a group of pieces of UI. Fig. 1 shows an overview of the relationships between the different entities: tasks, UI Blocks and FC part. As a result, to implement our model, we need to retrieve from all the composed applications both their tasks description and the links between parts of their UI and parts of their FC. This knowledge is represented in a so-called Enhanced Task Tree.

<figure><img src="image_88.png" /><figcaption>Fig. 1. Links between tasks, UI tree and FC</figcaption></figure>
## 3.2 Enhanced Task Tree Definition

We define three sets for each application:

* Let *UI* the set of UI blocks.

* Let *T* the set of tasks.

* Let *FC* the set of functionalities.

The sets FC, UI and T of an application are defined by analyzing this application with the aim of extracting knowledge to create an enhanced task tree (ETT). Such a task tree includes the description of UI (blocks and layout) and links with the Functional Core (FC). This analysis is performed by developers.

Based on the ontology presented in the right part of the **Fig. 1**, we represent an ETT by a knowledge graph linking the multiple conception levels: it captures the links between the tasks in *T* and the UI blocks in *UI*, and between the tasks in *T* and the functionalities in *FC*. A task may be linked to at least one FC or UI entities. A UI (and respectively a FC) is at least linked to one task.

With ETTs, we are able to extract the right part of the reused UI in order to place them in the new UI without losing the links with the FC. ETTs enable composition at different levels that we can express through functions. In the remainder of section 3, we present the composition functions we specified for each of the three levels.

## 3.3 Selection/Extraction of Tasks at UI level

To represent the links between information, we define three functions:

Let $\delta$ a function associating to a UI block the corresponding tasks and its inverse $\delta^{-1}$ associating to each task its corresponding UI blocks:

$$ \delta: UI \longrightarrow T^+; u \mapsto \{t_1, \dots, t_n\} $$

$$ \delta^{-1}: T \longrightarrow UI^+; t \mapsto \{u_1, \dots, u_m\} $$

In order to go further than the simple UI hierarchical relationship of container-component, we can identify each UI blocks {$u_k$} connected with a given UI block "u", i.e. {$u_k, \forall k, \exists j, t_j \in \delta(u)$} and $u_k \in \delta^{-1}(t_j)$. That connection is the expression that all UI blocks {$u_k$} are required to perform the tasks associated with the given UI block "u". So it makes sense to extend the selection from the single UI block "u" up to the set of UI blocks {$u_k$}, according to the acknowledgment of the developer.

## 3.4 Selection/Extraction of Tasks at Task level

Let $\rho$ a function associating to each task all the tasks related to it:

$$ \rho: T \longrightarrow T^+; t \longrightarrow \{t_1, \dots, t_n\} | \forall i \in \{1, \dots, n\}, r(t, t_i) $$

where *r* is a relation representing the temporal operations between tasks and the hierarchy between tasks from the CTT model [5].

Sometime, extracting only one task, like a dialog box to select a file, may have no sense, because the result of the task is not used. So if the developer has selected a task "t" through the UI blocks selection by selecting the set of UI blocks defined by
$\{u_k, \forall k, u_k \in \delta^{-1}(t)\}$, the developer has to extend that "t" up to $\rho(t)$. By this way, the developer can select some (or all) tasks directly related to "t". So, the function $\rho \circ \delta$ retrieves the tasks attached to a given UI block. The function $\chi$ representing the set of UI elements selected by extension of an initial selection:

$$ \chi: UI^+ \longrightarrow UI^+ \\ \chi(\{ui_1, \dots, ui_n\}) = \{ui_1, \dots, ui_p\} | p \ge n \land \forall ui_{k \in \{n+1, \dots, p\}}, \exists j \in \{1, \dots, n\}, \exists t \in \rho(\deltaui_j) | ui_k \in \delta^{-1}(t) $$

### 3.5 Selection/Extraction of Tasks at FC level

Let $\gamma$ a function associating to a task all its corresponding FC elements:

$\gamma: T \longrightarrow FC^+; t \longrightarrow \{fc_1, \dots, fc_n\} | \forall i \in \{1, \dots, n\}, c(t, fc_i)$ where c is a relation representing the links between a functionality and the task it implements.

Like $\rho$, $\gamma$ enables extension of selection, but at a functional level, i.e. at the data processing level. The retrieval of the functionalities attached to a given UI block relies on the functions $\gamma \circ \delta$ and $\gamma \circ \rho \circ \delta$.

## 4. Composition Process

The goal of the process is to produce a new application resulting from the composition of UI of former applications. The new UI is composed of several parts reused from former UI and possibly of some new consolidating parts like graphical glue used to fill remaining voids.

Through a special UI, the user (i.e. a developer) loads each application containing functionalities (and corresponding UI blocks) to be inserted in the new application. The loading step corresponds to the construction of the multiple level descriptions introduced above. From each application's UI, he selects UI blocks to be reused. Thus, he can compose his new application. He obtains in only one application the different functionalities he wants to keep from the reused applications.

The construction of the new UI is done in three steps iterated during the building of the complete new UI:

1. First, the developer makes a selection of pieces of UI to be reused in the new UI. Here we check whether the selected UI block is valid or not. To be valid, a UI block must enable the end user to completely perform one or more functionalities. He may either select an entire screen to add in the new UI, or select a UI block to be reused in the new UI, or select several UI blocks. In that case, the relative positions of the selected blocks in the initial UI are kept and they are placed in a new undividable UI block.

2. If the selected UI block is not valid, we propose the extraction of complementary pieces of UI to "validate" the selection. During this extraction step, questions are asked to the developer to help the validation of the UI block. This step constitutes the extension of the selection.
3. Once the selected UI block is valid, the third step consists in the placement of it in the new UI through various possible layouts. The selected UI is an entire screen, he has the possibility to place selected screen in the new screen flow. If it is a UI block, he can place it in the screen according predefined layouts that are proposed to the developer to define the placement in a screen or in a group of UI blocks.

# 5. Implementation: UI for Composing UI

We developed a proof of concept to perform a first validation of the different steps of the process we propose. It is made of (i) a UI to graphically compose several applications and (ii) several well-built applications. By “well-built application” we mean that it is developed with a clear separation between the UI and the Functional part. Moreover, both parts of the application are “public”: for the UI part, we can explore all UI Components starting from the main window (and its content pane); for the FC part, we get all the called methods, i.e. the Functional part is accessible through a “façade”. By “well-built application” we also mean that it is provided with an external description, its Enhanced Task Tree (ETT). The whole development is made in Java.

## 5.1 Enhanced Task Tree

ETTs are represented in RDF<fnref n="1" />, the W3C standard for the Semantic Web; the ontology is represented in RDFS<fnref n="2" />, the W3C standard for light-weight ontologies. To implement our composition mechanism, we use the Corese [1] semantic web engine to process and query the RDF representations of the different parts of the application. We implement the functions $\delta$, $\rho$ and $\gamma$ by SPARQL<fnref n="3" /> queries over the RDF(S) representation of the ETT.

Our model of tasks is based on CTT [5] and our UI model is based on MARIA [7] (we added some UI elements like graphical glue for the description of UI component tree). In our RDF models, there are references to Java Objects both for UI and FC. Thus, we define a unique ID for each UI component, based on the main class of the application and the place of the UI component in the component tree. For the Functional part, we define a unique ID based on its “façade” class and method name.

## 5.2 Selection and Extraction

We developed a UI for manipulating the different applications the user (i.e. a developer) wants to compose. This tool lets the developer compose his new

<footnote n="1">¹ http://www.w3.org/RDF/</footnote>

<footnote n="2">² http://www.w3.org/TR/rdfschema/</footnote>

<footnote n="3">³ http://www.w3.org/TR/rdf-sparql-query/</footnote>
application, place the different elements and save or load a composition already done. The developer performs the selection step by interacting with the former UI and by controlling those interactions with our tool for manipulating. Indeed, our tool enables to activate or deactivate the interaction in the former UIs (by adding / removing initial graphical event listeners) and the selection process (by adding / removing our own graphical event listeners)

The extraction step is interleaved with the selection step: each time a (group of) UI component(s) is selected, its extraction is determined as a set of questions asked to the developer. Because we have an access of the task tree corresponding to actions performed through the interface, we are able to warn the developer of the need of extracting other components linked to the selected component. The developer can deactivate the questioning.

## 5.3 Placement

For this step, we propose to place components between each others, through relative positions like “above of”, “on the right of”, “on the left of”, etc...

We express conditions with RDF properties and we transform these conditions in a Java layout. At the same time, the Corese engine deduces relative layouts thanks to a base of 14 inference rules we wrote. For example, from absolute positions of two different UI components, our rules enable Corese to deduce whether the first component is on the left, on the right, above or below the second component. This deduction is necessary to provide a relevant feedback to the developer during the placement step.

# 6. Conclusion

In this paper, we proposed an original process based on the manipulation of UI that improves the composition result in terms of UI design reuse while preserving the links between FC and UI parts.

Our process is made of three steps: selection, extraction and placement of former UI blocks. Each of these steps uses the enhanced task trees associated to the applications to compose and to build a new task tree keeping some links between the parts of applications. The originalities of the proposed process are: (i) in its starting point (the UI) but with a cover of also Task Model and Functional Core; (ii) in our commitment to reuse former UI (including their design properties). Moreover, its strengths are: (iii) in the possibility to build the resulting enhanced task tree in function of the user actions on the former UI and (iv) in the extraction of the right part of the UI and its placement in the new UI without losing the links with the FC.

Our approach must to be improved before performing test with developers. Indeed, we are working on merging UI blocks, at different levels (FC, TM or UI), according to the compatibility of manipulated entities and by importing adapters given by the developer.
# 7. Acknowledgments

Our work is funded by the DGE M-Pub 08 2 93 0702 project.

# References

1. Corby O., Dieng-Kuntz R., and Faron-Zucker C. Querying the semantic web with the corese search engine. In 16th European Conference on Artificial Intelligence (ECAI2004), IOS Press, Valencia, Spain, 2004.

2. Gabillon Y., Calvary G., and Fiorino H. Composing interactive systems by planning. In UbiMob'08, pages 37–40, Saint Malo, France, 28 - 30 mai 2008.

3. Lepreux S., Hariri A., Rouillard J., Tabary D., Tarby J.-C., and Kolski C. Towards multimodal user interfaces composition based on usixml and mbd principles. Lecture Notes in Computer Science, 4552(134):134–143, July 2007.

4. Lewandowski A., Lepreux S., and Bourguin G. Tasks models merging for high-level component composition. Human-Computer Interaction, Part I, HCII 2007, Lecture Notes in Computer Science (LNCS), 4550:1129–1138, July 2007.

5. Mori G., Paternò F., and Santoro C. Ctte: Support for developing and analyzing task models for interactive system design. IEEE Transactions on Software Engineering, pages 797–813, Aug. 2002.

6. Occello, A., Joffroy, C., Pinna-Déry, A.-M., Renevier, P. and Riveill, M. Experiments in Model Driven Composition of User Interfaces. In 10th IFIP International Conference on Distributed Applications and Interoperable Systems (DAIS'10), volume LNCS 6115, pages 98-111, Amsterdam, Netherlands, 2010. Springer-Verlag.

7. Paternò F., Santoro C., and Spano L. D.. Maria: A universal, declarative, multiple abstraction level language for service-oriented applications in ubiquitous environments. In Computer-Human Interaction (TOCHI), volume 16, Nov 2009.

8. Pinna-Déry A.-M. and Fierstone J. Component model and programming: a first step to manage Human Computer Interaction Adaptation. In Mobile HCI'03, volume LNCS 2795, pages 456–460, Udine, Italy, Sept. 2003. L. Chittaro (Ed.), Springer Verlag.

9. Tsai W.-T., Huang Q., Elston J., and Chen Y. Service-oriented user interface modeling and composition. In ICEBE '08, pages 21–28, Washington, DC, USA, 2008. IEEE Computer Society.

10. Grundy J.C. and Hosking J.G. Developing Adaptable User Interfaces for Component-based Systems. In Interacting with Computers, Volume 14, 2, pages 175-194., March 2002, Elsevier Science Publishers.

11. Fujima, J., Lunzer, A., Hornbæk, K. and Tanaka, Y. Clip, Connect, Clone: Combining Application Elements to Build Custom Interfaces for Information Access, In Proceedings of UIST 2004, pages 175-184, Santa Fe, NM, 2004.

12. Elkoutbi, M., Khriss, I., and Keller, R.K. Generating User Interface Prototypes from Scenarios, In RE'99, Fourth IEEE International Symposium on Requirements Engineering, pages 150-158, Limerick, Ireland, June 1999.

13. Ginzburg, J., Rossi, G., Urbieta, M. and Distante D. Transparent interface composition in Web Applications, In proceedings of Web Engineering, Volume 4607, pages 152-166, Heidelberg, 2007, LNCS, Springer
# Methods for efficient development of task-based applications

Vaclav Slovacek¹,

¹ Dept. of Computer Graphics and Interaction, Faculty of Electrical Engineering, Czech Technical University in Prague, Karlovo nám. 13, 121 35, Praha 2, Czech Republic
slovavac@fel.cvut.cz

**Abstract.** This paper introduces methods for developing task-based applications by tightly integrating workflows with application logic written in an imperative programming language and automatically completing workflows especially with tasks that mediate interaction with users. Developers are then provided with completed workflow they may be used for further development. Automatic completion of workflows should enable to significantly shorten the development process and eliminate repetitive and error-prone development tasks. Information extracted from workflow structure and low level application logic may then be used to automatically generate low to high fidelity prototype user interfaces for different devices and contexts.

**Keywords:** Workflow, workflow processing, task modeling, generated user interface

## 1 Introduction

The main goal of the research related to this paper is to introduce methodology that would enable efficient development of task-based applications using visual task modeling and present functionality of an ongoing task framework implementation.

Developing applications by first designing a task model and then automatically generate user interface provides developers with option to model application on higher levels of abstraction as task models abstract from device display resolutions, input methods, available user interface components, etc.

Modeling an application using workflows provides formal description of all processes in the application that is understandable by non-programmers. It enables to design and analyze the application on different levels of abstraction (using nested workflows) and enables eventually to detect design issues in the processes design [1].

Despite all the advantages listed above, current methods also suffer from significant drawbacks that prevent many developers from adopting task modeling as a method for developing applications. Developers have to learn formal semantics and although workflow schemas are easy to read they are much harder to design properly, especially when different tasks may run in parallel, cancel each other, etc.

Workflows may be used to automatically generate user interfaces (for example web forms that asks user for required input) and we believe that by making designing workflows simpler the workflows may be then used at least for rapidly designing low
fidelity prototypes that may be modified by user interface designers, used for usability
testing and finally converted to final user interfaces accelerating development cycle.

## 2 State of the art

Common tool for task modeling are ConcurTaskTrees [2] that are used for
hierarchical task analysis [3]. ConcurTaskTrees describe a high level task by
hierarchically splitting it into subtasks. Abstract tasks that represent users intentions
are split into elementary tasks. These tasks are either machine tasks (performed by a
device a user is interacting with), user tasks (performed by a user) and interaction
tasks (user interacting with the device). Complex branched processes are easier to
express using workflow languages such as YAWL [4] and BPEL [5] that enable do
define branching, iterations, etc. There are also several implementations of a
workflow engines and visual editors available for both YAWL and BPEL.

There are several projects focused on automatic user interface generation. Project SUPPLE[6] currently enables to automatically generate user interfaces. It is possible to provide SUPPLE with data that are required to be entered by a user (specifying the type of data, name, allowed and expected values) and SUPPLE provides a user interface optimized for a specific user [7] enabling the user to enter the required data.

## 3 Framework

We focused on making the process of designing task-based applications as usable for
developers as possible. We took advantage of currently existing technologies and built
on knowledge of developer enabling them to quickly start developing more
maintainable applications while providing methods for rapidly deliver low to high
fidelity user interface prototypes that are integrated with application logic.

### 3.1 Workflow

We have chosen YAWL as a base language for our research because it covers all
necessary patterns for describing any application logic unlike BPEL [4]. Also every
BPEL process can be converted to an appropriate YAWL workflow [8]. We use a
simplified YAWL notation closely described in [9] that reduces number of visual
elements to describe a workflow.

We also do not use XPath and XQuery that are used in YAWL for branching
conditions and data updates. Instead we delegate branching logic to an imperative
programming language such as Java and provide an API for controlling execution of
the workflow (e.g., choosing tasks should be executed). Using an imperative
programming language for elementary application logic should be natural for most
developers.
## 3.2 Task types

Workflow is a descriptive form of defining tasks. Although workflows are efficient for describing high level processes, it generally fails to simply describe low-level application logic [10] that is much more efficiently described using imperative programming languages such as Java, ECMAScript, etc. By low-level application logic we understand operations that have no inner structure that should be exposed.

The problem with using an imperative language for implementing elementary tasks is that it is very hard to limit their functionality so it does not perform operations that should be rather defined in a workflow. Decision what is still considered elementary task depends on developers and thus a set of rules and recommendations for proper coding style should be introduced. Otherwise advantages of having a descriptive task-model may be lost as most of the application logic might be implemented in elementary tasks that have no inner structure and behave as black boxes.

We split tasks into implicit, triggered and user task types. These types differ by when they are executed. Implicit tasks are executed by the workflow engine automatically when they are reached in an executed workflow. Triggered tasks must be initialized from a user interface and are easily recognizable as they require an input data. Server stops processing workflow until it receives a triggering event with the required data from a user interface.

User tasks are similar to triggered tasks, but are directly accessible by user, so there must be a button or another interaction element visible that enables users to initialize the task. Developer is responsible for declaring user tasks to distinguish them from triggered tasks (e.g., using `@UserTask` annotation). This information is important for automatically generating user interfaces.

## 3.3 Execution conditions

Execution conditions are conditions that must be satisfied for a task to be available for user. The conditions in our framework are identified by a unique id (e.g., fully qualified name of class that is used to evaluate if the condition is satisfied).

As the framework abstracts from how execution conditions are represented. It is possible to extend the support of execution conditions to Java Bean Validation [11], semantically described conditions, etc.

## 3.4 Execution condition satisfiers

Execution condition satisfier is a task that is automatically inserted before another task to satisfy its execution conditions. The condition satisfier itself may require different execution conditions to be satisfied and thus it might be necessary to add another condition satisfier preceding it (e.g., condition satisfier producing a user object instance, may require user name and password).

The framework abstracts from how a condition satisfier task satisfies an execution condition so different implementations may be used - e.g., implementation providing manually designed user interface to a user, implementation providing automatically generated user interfaces, implementation providing static data, etc.
# 4 Modifying workflow

To simplify and accelerate workflow-based application development we propose a method for modifying workflow that completes the workflow automatically based on execution conditions extracted from source code of elementary tasks.

An application is a set of tasks the user may perform (the tasks the application was designed to perform). These tasks are usually implemented in elementary tasks and the rest of the workflow ensures they are performed in the desired order, under certain conditions (e.g., time, location), after required data get available, etc.

## 4.1 Satisfying execution conditions

The figure 1 shows four different representations of the same workflow. The workflow as defined by developer (a) contains just one task that requires a condition uniquely identified by the letter A (shown in the circle on the left side of the task) to be satisfied.

<figure><img src="image_89.png" /><figcaption>**Fig. 1.** The same task represented in 4 different ways - a) as defined by a developer, b) automatically completed, c) simplified semantic representation that is more readable by a developer, but not is semantically equivalent with (b), d) wrapping task with complex inner structure in form of nested workflow</figcaption></figure>

The framework modifies the workflow (b) by adding a task (Display UI) that renders a user interface visible to user and then the execution stalls before executing task Retrieve Data waiting for user input (tasks requiring input from a user interface are marked by a thicker border). After a user provides the required data the original task is executed in parallel with another added tasks that notifies user about progress of the original task which may be important for user experience [12] (e.g., displaying a progress bar, showing notification when the task is finished, etc., depending on implementation).

Although the automatically completed workflow exactly represents the application behavior it is quite complex and adding such constructs to every task with unsatisfied execution conditions would lead to overly complex structure that might be difficult to work with.

Because of this the framework might represent the task by merging all the added tasks into a single task (User Interface) that precedes the original task as shown in (c), this however leads to losing the parallel branch that lets user informed about progress of the task.

The only acceptable pattern seems to be creating a wrapping task that encapsulates a nested workflow similar to the one shown in (b). As all execution conditions are
solved inside the nested workflow there is no unsatisfied execution condition on the higher level and the higher level workflow is kept simple compared to (b) and (c).

## 4.2 Completing workflow

Satisfying an execution condition a single task is useful for dealing with very simple patterns. Eventually satisfying execution conditions for each task individually may lead to generating poor user interfaces.

The figure 2 shows an example workflow having two sequentially executed tasks (a). Using the method described in the previous section the framework would provide two separate user interfaces preceding each task. However there is no other branching or conditional logic that would cancel the sequential execution thus the execution conditions may be collected and a user interface preceding execution of the first task may be provided asking user for input required for both tasks as shown in (b).

A different pattern is shown in (c) when a common user interface is used to get input necessary for both task and then asks for additional input necessary only for one of the branches (d) while the other branch may be executed while waiting for a user to input necessary data.

<figure><img src="image_90.png" /><figcaption>Fig. 2. Example workflows as defined by a developer (a), (c) and with automatically added condition satisfier tasks (b), (d)</figcaption></figure>

# 5 Metadata related to user interfaces

Information necessary for rendering user interfaces is directly extracted from elementary task source code. A lot of information may be extracted from the source code using language reflection, which is supported by several programming
languages[13], without a developer having to provide any additional information. The
following Java code is an example of an elementary task as used in our framework:

```java
@Title("title") @Description("description")
class Login implements UserInteraction {
    void execute(
        @NotNull DeviceContext requiredDeviceContext,
        @Title("title") @Description("description")
        @Input("username") String name,
        @Input("password") String password) {
        // application logic here
    }
}
```

The above code snippet contains enough information to be properly represented in a user interface. The framework extracts fields of the class and parameters of the `execute()` method. Based on their types and names defined in `@Input` annotation it injects appropriate values to class fields using inversion of control mechanism before calling the `execute()` method with appropriate parameters.

The class implements interface `UserInteraction` that extends interface `Interaction` common for every elementary task. As interfaces support inheritance they may be used to organize interactions into different groups. This may be used for better layout of automatically generated user interface (e.g., clustering of related user interface elements).

Additionally the code snippet above contains annotations (`@Title` and `@Description`) that provide closer description of both the interaction itself and the parameter expected to be provided by a user. These information may be used to properly represent the task in a user interface. We have introduced annotations that are a copy of Dublin Core [14] metadata tags used typically in XML to describe these properties of elementary tasks. The title may be for example used for a label related to a text field for input of the appropriate data and description for providing a tooltip closer describing the required input.

## 5.1 Task importance

There are several ways how importance of tasks for a given application may be calculated. The framework enables to specify the importance of certain tasks manually (e.g., using `@Importance` annotation). This is a mechanism that tells which tasks are necessary in the application. This typically includes the tasks the application is designed for (e.g., volume control and channel switching in a TV application).

However specifying importance manually to all tasks is not very useful. Thus importance may also depend on user preferences (for example unexperienced user will not access color management settings of a TV), on an end-device used or any other environment properties.

Calculating transitive importance is used to propagate importance of individual tasks forward in the workflow. We simply do that by propagating the highest importance to previous tasks in a sequence. This ensures that the task is properly represented in the interface that is provided to user before it is even reached in a
workflow. An example of the importance propagation is shown in figure 3 with
workflow (b) showing transitive importance calculated from workflow (a) and
workflow (d) showing transitive importance from workflow (c). In workflow (d) the
importance is propagated only to a single branch that already has higher importance
to prevent cluttering a user interface.

<figure><img src="image_91.png" /><figcaption>**Fig. 3.** Workflow with tasks with calculated importance (a) parallel tasks with individually calculated importance, (b) the workflow with transitive calculated importance, (c) sequential tasks with individually calculated importance, (d) the same workflow with transitive calculated importance</figcaption></figure>

Calculated importance may then be used also for automatically generating user interfaces where more important tasks may be represented e.g., by larger buttons, more prominently placed controls, etc. Tasks with importance under a defined threshold may eventually not be visible in a very constrained environment (e.g., very small device screen).

We distinguish three different states of a workflow task. Task may be enabled (all execution conditions are satisfied) meaning that user may execute it. Task may be disabled (at least one execution condition is not satisfied) causing all relevant user interface elements (e.g., button executing the task) to be disabled but still visible in the user interface thus keeping it consistent. Tasks that are in hidden state are not represented in the user interface and their controls simply disappear. As it would be difficult for a computer system to guess whether the tasks should be disabled or hidden developer has to declare that a task switches to disabled state (e.g., by adding @AlwaysVisible annotation).

# 6 Conclusion

We have introduced basic methods for automatically completing workflow models for developing applications that should simplify development of applications based on workflows.

We have strongly focused on process that would enable to automatically generate user interfaces directly from application logic implemented in an imperative programming language while managing the state transitions based on workflow description tightly integrated with an application low-level source code.
# 7 Future research

We have not covered a situation that may occur when an execution condition is first satisfied by condition satisfier task, then concurrently made unsatisfied by another tasks and then a tasks requiring the execution condition to be satisfied is reached.

It would be possible to enclose blocks that depend on satisfying execution conditions into transactions. However as these transactions might involve user interaction there may arise problems with very long transactions blocking other processes in an application and problems with selecting proper items in a context to be locked. Transactions locking large part of a context for a long time may result in deadlock and/or may significantly slow down an application.

Research should also focus on usability of automatically generated user interfaces and analyze impact of methods described in this paper on development process.

# References

1. Van Breugel, F. and Koshkina, M. Models and Verification of BPEL. *Unpublished Draft* (Jan 1 2006).

2. Paternò, F., Mancini, C. and Meniconi, S. *ConcurTaskTrees: A diagrammatic notation for specifying task models*. Chapman & Hall, Ltd. London, UK, UK, City, 1997.

3. Stanton, N. Hierarchical task analysis: Developments, applications, and extensions. *Applied Ergonomics* (Jan 1 2006).

4. Van der Aalst, W. and Ter Hofstede, A. YAWL: yet another workflow language. *Information Systems*, 30, 4 2005), 245-275.

5. Juric, M. B. Business Process Execution Language for Web Services BPEL and BPEL4WS 2nd Edition (Jan 1 2006).

6. Gajos, K. and Weld, D. SUPPLE: automatically generating user interfaces. *Proceedings of the 9th international conference ...* (Jan 1 2004).

7. Gajos, K., Wobbrock, J. and Weld, D. Automatically generating user interfaces adapted to users' motor and vision ... *symposium on User interface ...* (Jan 1 2007).

8. Brogi, A. and Popescu, R. From BPEL processes to YAWL workflows. *Lecture notes in computer science*, 41842006), 107-122.

9. Slovacek, V. Towards Workflow-based Application Development Framework (Jan 1 2010).

10. Jelinek, J. and Slavik, P. GUI generation from annotated source code. *Proceedings of the 3rd annual conference on ...* (Jan 1 2004).

11. Bernard, E. and Peterson, S. JSR-303 Bean Validation. *Bean Validation Expert Group* (Jan 1 2009).

12. Blackmon, M. Cognitive walkthrough. *Encyclopedia of human-computer interaction*.

13. Gosling, J., Joy, B., Steele, G. and Bracha, G. *Java (TM) Language Specification, The (Java (Addison-Wesley))*. Addison-Wesley Professional, 2005.

14. Weibel, S., Kunze, J., Lagoze, C. and Wolf, M. RFC2413: Dublin Core Metadata for Resource Discovery. *RFC Editor United States* (Jan 1 1998).
# Towards an Integrated Model for Functional and User Interface Requirements

Rabeb Mizouni<fnref n="1" />, Daniel Sinnig<fnref n="2" />, Ferhat Khendek<fnref n="3" />,

mizouni@uaeu.ac.ae

dasin@informatik.uni-rostock.de

khendek@encs.concordia.ca

**Abstract:** Despite the widespread adoption of UML as a standard for modeling software systems, it does not provide adequate support for specifying User Interface (UI) requirements. It has become a common practice to separately use UML use cases for specifying functional requirements and task models for modeling UI requirements. The lack of integration of these two related models is likely to introduce redundancies and inconsistencies into the software development process. In this paper, we propose an integrated model, consisting of use case and task models, for capturing functional and UI requirements. Both artifacts are used in a complementary manner and are formally related through so-called *Anchors*. Anchors are use case steps that require further elaboration with UI-specific interactions. These interactions are explicitly captured in associated task models. The formal semantics of the integrated model is given with finite state automata.

**Keywords:** Functional Requirements, UML Use Cases, User Interface Requirements, Task Models, Integrated Requirements Model, Finite State Automata.

## 1 Introduction

UML has become the de-facto standard for software systems modeling. However, UML's support for User Interface (UI) development is deemed insufficient [1]. While UML diagrams are well suited for object-oriented analysis and design, the HCI community argues that a set of specialized models is needed to effectively specify users' characteristics and tasks, UI dialogue structures and layouts.

This divergence has been addressed by many researchers. Most attempts either define extensions for UML to capture HCI related information [2, 3] or, conversely, extend HCI models to cope with object-oriented features [4, 5]. An effective integration, however, is not simply a matter of expressiveness and the ability to convert or embed a model into another one. Instead, as Paternò [1] points out, specialized notations should be used in a complementary manner to efficiently support software engineers and UI designers in their work.

In this paper, we define an integrated model for capturing functional and UI requirements. It is composed of two heterogeneous, yet interrelated parts: *UML use cases* and *HCI task models*. Use cases are the medium of choice for capturing functional requirements whereas task models are commonly used to specify the

<footnote n="1">¹College of Information Technology, UAE University, Al-Ain, UAE.</footnote>

<footnote n="2">²Faculty of CS and Elec. Engineering, University of Rostock, Germany.</footnote>

<footnote n="3">³Dept. of Electrical & Computer Eng., Concordia University, Montreal, Canada.</footnote>
<figure><img src="image_92.png" /><figcaption>Fig. 1. Use Case Model Syntax with Anchor</figcaption></figure>

detailed user interactions with the system. Within our integrated model, use cases and task models are used according to their intended purposes establishing clear separation of concerns.

The research reported here builds upon our earlier work [6, 7] where we described a two-phase integrated development methodology for use cases and task models. In the *first* phase, an initial coarse grained use case model is developed, which, without delving into details, documents the primary interactions that actors will perform with the system in a step-by-step format. Additionally, for each use case, the software engineers identify a set of use case steps that require further elaboration with UI details. These steps are called *anchors*. In the *second* phase, each anchor is associated with a corresponding task model capturing UI-specific interactions. Concurrently, the coarse-grained use case model is further refined by taking into account alternative and failure cases that thus far have been considered only marginally.

In this paper we focus on the definition of an integrated model for functional and UI requirements to support such a methodology, including its syntax and semantics. The latter is defined by providing a formal mapping to the semantic domain of finite state automata.

## 2 Syntax of the Integrated Functional and UI Requirements Model

In this section we define the syntax of our integrated model for functional and UI requirements. As aforementioned, the model consists of two heterogeneous parts, a use case model and a set of task models, interlinked by a set of anchor points. Each individual use case corresponds to the structure portrayed in Fig. 1. The main success scenario as well as each extension consist of a sequence of use-case steps, which can be of six different kinds. *Atomic* steps are performed either by the system or a secondary actor. They contain a textual description, but do not consist of any sub-steps. *Anchors* are also atomic, but are performed by the primary actor and as such are related to the user interface. Anchor steps additionally contain a reference to a refining (UI-specific) task model. *Choice* steps provide the primary actor with the choice between several interactions. Each such interaction is (in turn) defined by a
<figure><img src="image_93.png" /><figcaption>Fig. 2. Integrated Functional and UI Requirements Model of the “Process Contact Request” Use Case</figcaption></figure>

sequence of steps. *Concurrent* steps define a set of steps which may be performed in any order by the primary actor. *Goto* steps denote jumps to steps within the same use case. *Include* steps denote invocations of sub-use cases.

To illustrate our approach, let us consider a “Process Contact Request” use case. It depicts the interactions involved in processing contact requests, as it is typical in social networks such as LinkedIn and Facebook. The main success scenario describes the situation in which the primary actor directly accomplishes his goal of confirming a contact request. We also define two extensions to specify alternative scenarios, which occur when the primary actor fails to authenticate himself or refuses a contact request, respectively. The textual description of the use case is shown on the left hand side of Fig. 2.

The use case contains two UI-related steps 1 (“Authentication”) and 4 (“Identification of Contact Request”) which are defined as anchors and as such are related to refining task models. Both steps do not detail how the step-goals are achieved. These interactions are UI-specific and are captured in the corresponding task models. For example, the authentication step (CuAu) may require that the user enters his/her name and password in any order (Desktop UI), or that the user dictates his/her login information (Text-Free Voice UI). Both possibilities are expressed by the binary choice operator ([]) in the corresponding task model (CuAu-TM). In a similar manner, use case step 4 (ConfReq) is associated with a task model (ConfReq-TM), specifying UI interactions for confirming a contact request.

## 3 Semantics of the Integrated Functional and UI Requirements Model

This section defines a formal semantics for our integrated model. We start by defining the well-known semantic domain of finite state automata. We then portray how the use case and task model parts of the integrated model are mapped separately into the
semantic domain. Finally, we define a merging procedure that integrates the various
individual semantic representations into a common behavioral model.

**3.1 Semantic Domain**

The semantics for our integrated model is given by a mapping to a finite state automaton.

**Definition 1 (Finite state automaton).** An automaton is defined as a 5-tuple $(S, s^0, S^f, L, E)$ where $S$ is the set of states, $s^0$ is the initial state, $S^f$ is the set of final states, $L$ is the set of labels, and $E \subseteq S \times L \times S$ is the set of transitions.

**Definition 2 (Trace).** A trace of an automaton $A$ is a sequence of transitions $e=q_0.q_1.q_2...q_{n-1}$ where $q_0=(s_0, l_0, s_1) \in E$ such that $s_0=s^0$, $\forall i, 1 \le i < n-1$ $q_i=(s_i, l_i, s_{i+1}) \in E$, and, $q_{n-1}=(s_{n-1}, l_{n-1}, s_n) \in E$ where $s_n \in S^f$

Informally, a trace is a word of the language accepted by the finite state automaton when it starts from its initial state and ends in one of its final states for the trace. In what follows, we use operational semantics for our definitions. Equations of the following form $\frac{a;b}{c} (Cond)$ denote that $a$ AND $b$ IMPLY $c$. $Cond$ is the condition for the applicability of the rule.

**3.2 Semantics for Use Cases and CTT Task Models**

This section outlines the separate mappings of the use case and task model into the semantic domain. For the sake of conciseness, only a high-level overview will be given while the full details can be found in [8].

The semantic mapping from a use case model into an automaton is defined in a bottom-up manner, starting with the mapping of individual use case steps. Each of the six kinds of use case steps enumerated in Fig.1 has its own specific mapping to an FSM. *Atomic* steps and *Anchor* steps map to elementary FSMs consisting of only an initial state and a set of final states, connected by a transition that represents the use case step. A *Choice* step maps to a composite FSM consisting of the initial states of each choice's FSM. A *Concurrent* step is the product machine of its constituent FSMs. *Goto* steps map to an FSM with a single state defined to be equivalent to the initial state of the FSM representing the target of the jump. The complement FSM of an *Include* step consists of two states: one identified with the initial state of the FSM of the main success scenario of the invoked sub-use case, and the other identified with all final states of the sub-use case's FSM.

Now that individual use case steps can be formally represented by automata, we can link arbitrary sequences of steps using sequential composition, by unifying the final states of the first operand with the initial state of the second one. In the next step, we map the main success scenario and each extension of the use case to a set of automata, each being the result of the sequential composition of the automata representing the individual use case steps. Finally, the entire use case is mapped into an automaton, by merging the automata representing the main success scenario and all its extensions.
Similar to the semantic mapping of use cases, the mapping of CTT task models to automata is performed in a bottom-up manner. Each atomic task is mapped into an atomic automaton. Composite tasks are represented by more complex automata, which result from the composition of the automata representing sub-tasks. We have defined the following composition operations: sequential composition (●), choice composition (#), parallel composition (||), and iterative composition (*). The full details of the mappings are given in [8].

### 3.3 Automaton of the Integrated Model

In this section, we elaborate how the individual use case and task model automata are merged into a single automaton, representing the behavioral semantics of our integrated requirements model.

Intuitively, the behavior of the integrated requirements model can be summarized as follows: At first, the integrated model adopts the behavior of the use case model up until an *Anchor* step is encountered. At this point, the integrated model adopts the behavior of the associated CTT model depicting how the primary actor may accomplish the step-goal using a particular UI. Thereafter, the integrated model again resumes with the behavior of the use case model. This alternating continues until the scenario comes to an end.

The behavioral merge of finite state automata has been addressed in many research projects [6, 9-11]. Since in our integrated model use cases and task models are utilized in a complementary-non-overlapping manner we choose one of the existing explicit automata composition techniques [6, 10, 11] to merge the respective use case and task model automata. Similar to our work presented in [6], the merge of use case and task model automata is based on imperative expressions. Each expression specifies (1) the use case and the CTT automata to be merged, (2) the anchor where the merge is performed, and (3) a *Refine* operator that specifies how the actual merge is performed. The evaluation of the expression yields a new automaton where the use case transition representing the anchor step has been replaced by the corresponding CTT automaton. We define *Refine* operator semantics next.

Let $A=(S,s^0,S^f,L,E)$ be an automaton and let $tr(A)=\{e \mid e \text{ is a trace of A}\}$ be its set of traces. We define the set $tr(A, ep)$ to be the set of traces of A passing through the anchor ep as: $tr(A, ep)=\{e \in tr(A), e= q_0.q_1.q_2....q_{n-1}, \forall 0 \le i \le n-1 q_i \in E \mid \exists q_i = ep\}$. It represents the set of traces where ep appears as a transition in the trace. Additionally, we define $Pref(A, ep)$ (respectively postfixes $Post(A, ep)$) as the set of prefixes (respectively postfixes) of the traces of the automaton A passing through ep. More formally, let $e=q_0.q_1.q_2....q_{n-1}$ be a trace of automaton A. $u=q_0.q_1....q_{i-1} \in Pref(A, ep)$ if ($q_i = ep$) (respectively, $r=q_{i+1}.q_{n-1} \in Post(A, ep)$ if ($q_i = ep$)). Consequently, a trace $e \in tr(A, ep)$ can be written as: $e=u.ep.r$ where $u \in Pref(A, ep)$ and $r \in Post(A, ep)$.

**Definition 4 (Refine Operator):** Let $A=(S_1,s_1^0,S_1^f,L_1,E_1)$ and $B=(S_2,s_2^0,S_2^f,L_2,E_2)$ two automata, and $C=(S_3,s_3^0,S_3^f,L_3,E_3)$ be the resulting automaton by applying the *Refine* operator at the anchor $t=(s,a,s')$. Furthermore, let $tr(A)$ and $tr(B)$ be the traces
of automata A and B, respectively. Then, the set of traces tr(C) of the resulting automaton C is constructed using the following rules:

$$ (1) \frac{(e \in tr(A) / tr(A,t))}{e \in tr(C)} $$

$$ (2) \frac{(e \in tr(A,t); (\exists u, r | e = u.t.r \text{ where } u \in \text{Pr ef}(A,t), r \in \text{Post}(A,t)); e_b \in tr(B))}{u.e_b, r \in tr(C)} $$

Equation (1) shows that all traces of A not passing through the anchor point are traces of the automaton C. Equation (2) shows that for all traces of A passing through t, the transition t is replaced by the traces of B.

The construction the final merged automaton is an iterative process, where the resulting automaton from a composition is used as an input to a subsequent composition until a fixpoint is reached (i.e., all anchors have been replaced by respective task model automata). At the end of the composition, the derived automaton is the semantic representation of the integrated requirements model. Fig.3 portrays the various automata involved in our "Process Contact Request" example. The automaton representing the use case is given in Fig. 3 (a). The task model automata representing the refinements of the "authentication step" and the "confirm contact" step are given in Fig. 3 (b) and Fig. 3 (c). Finally, Fig. 3 (d) illustrates the resulting automaton representing the integrated model.

## 4 Related Work

Since UML was developed with little attention to UI related issues, several proposals have been put forward to close this gap. Most of them fall into one of the following categories: (1) Extensions to UML for the purpose of capturing HCI-related information [2, 3], (2) extensions to HCI models for capturing object-oriented features [4, 5] and (3) development methods promoting the integration of HCI and OO models [12].

Paternò [1] proposes a method for integrating use case diagrams and task models. Use cases denote core functionalities offered by the system which are refined by a set of task models. However, the scenario descriptions entailed in each use case are not taken into account. Another approach that falls under the first category is presented by Noberga et al. [2]. Motivated by the fact that the current UML standard provides insufficient support for modeling interactive systems, a mapping from CTT task models to UML activity diagrams is proposed. The mapping is complemented by an extension of UML with high-level syntactic constructs related to task modeling.

Da Silva and Paton [5] propose UMLi as a modeling language for interactive systems. UMLi extends UML with UI diagrams for describing abstract interaction objects. According to their eight-step methodology, use cases are employed to define high-level functionalities which are further refined by a set of user tasks captured in extended UML activity diagrams. A set of logical links, placed between the various use cases and the activity diagrams, establishes traceability between UI details and the corresponding functional requirements.
<figure><img src="image_94.png" /><figcaption>Fig. 3. Composition Example</figcaption></figure>

Rosson [13] proposes a scenario-based approach to object-oriented analysis and design. In order to integrate usability concerns with functional modeling, a system is modeled by a set of *instance scenarios*. In a bottom-up approach the various scenarios are processed and serve as a basis for the creation of the object model. Nunes and Conha [4] point out that UML provides inadequate support for modeling architectural concerns of interactive systems and propose their Wisdom framework to fill this gap. While mainly based on existing UML models, Wisdom introduces a CTT-like notation to capture the dialogue between users and the application.

# 5 Conclusion

In order to overcome the insufficient support for UI modeling in UML, we have proposed an integrated model to capture functional and UI requirements. This integrated model is the outcome of a larger undertaking, first discussed in [7], that investigates methods for efficient collaboration between software engineers and UI designers while preserving clear separation of concerns. The integrated model is comprised of two well-established models – UML use cases and CTT task models – interrelated through a set of Anchors. We have defined a formal syntax and semantics for the integrated model. The latter is given in terms of a finite state automaton.

As future work, we plan to carry out comprehensive case studies and to apply our approach and notation to industrial-strength projects. We are currently developing tool support for authoring and validating the integrated model. We envision that our
tool will support the reuse of task model specifications (either within the same project or among different projects). In many cases, the interactions specified by task models are independent from the application domain and consequently can be reused across projects. Other future avenues are related to the extension of the integrated model to encompass other UI-related artifacts such as user and dialogue models and the generation of integrated test cases.

## References

1. Paternò, F. and C. Santoro, *Support for Reasoning about Interactive Systems through Human-Computer Interaction Designers' Representations*. Comput. J., 2003. 46(4): p. 340-357.

2. Nobrega, L., N.J. Nunes, and H. Coelho, *Mapping ConcurTaskTrees into UML 2.0*. Gilroy, S.W., Harrison, M.D. (eds) Interactive System, 2006. 3941.

3. Bastide, R.e., *An Integration of Task and Use Case Metamodels*. HCI International, San Diego, CA, USA, 19/07/09-24/07/09, 2009.

4. Nunes, N.J. and J.o.F. o e Cunha, *Towards a UML profile for interaction design: the Wisdom approach*. 2000.

5. de Paula, M.i.G., B.S. da Silva, and S.D.J. Barbosa, *Using an interaction model as a resource for communication in design*. CHI '05: CHI '05 extended abstracts on Human factors in computing systems, 2005: p. 1713-1716.

6. Mizouni, R., et al., *Merging partial system behaviors: composition of use-case automata*. IET Software, 2007. 1(4): p. 143-160.

7. Daniel, S., M. Rabeb, and K. Ferhat, *Bridging the gap: empowering use cases with task models*, in *Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems*. 2010, ACM: Berlin, Germany.

8. Sinnig, D., *Use Case and Task Models: Formal Unification and Integrated Development Methodology*. 2008, Department of Computer Science and Software Engineering, Concordia University, Montreal.

9. Chechik, M., et al., *Partial Behavioral Models for Requirements and Early Design*. MMOSS, 2006. 06351.

10. S.Leue, L. Mehrmann, and M. Rezai, *Synthesizing ROOM Models from Message Sequence Chart Specifications*. Technical Report 98-06, ECE Dept., University of Waterloo, Canada, october 1998.

11. Uchitel, S., J. Kramer, and J. Magee, *Behavior Model Elaboration using Partial Labeled Transition Systems* ESEC/FSE 2003, 2003.

12. Lu, S., et al., *Generating UML Diagrams from Task Models*. 2003.

13. Rosson, M.B., *Integrating development of task and object models*. Commun. ACM, 1999. 42(1): p. 49-56.