<figure><img src="image_1.png" /></figure>

HAL
open science

# Model Assisted Creativity Sessions for the Design of Mixed Interactive Systems: a Protocol Analysis

Christophe Bortolaso, Emmanuel Dubois

<figure><img src="image_2.png" /></figure>

## To cite this version:

Christophe Bortolaso, Emmanuel Dubois. Model Assisted Creativity Sessions for the Design of Mixed Interactive Systems: a Protocol Analysis. 14th International Conference on Human-Computer Interaction (INTERACT 2013), Sep 2013, Cap Town, South Africa. pp.126-143, 10.1007/978-3-642-40477-1_8. hal-01264520

HAL Id: hal-01264520

https://hal.science/hal-01264520

Submitted on 29 Jan 2016

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Open Archive TOULOUSE Archive Ouverte (OATAO)

OATAO is an open access repository that collects the work of Toulouse researchers and makes it freely available over the web where possible.

This is an author-deposited version published in : http://oatao.univ-toulouse.fr/
Eprints ID : 12424

The contribution was presented at INTERACT 2013 :

http://www.interact2013.org/

Official URL: http://dx.doi.org/10.1007/978-3-642-40477-1 8

**To cite this version** : Bortolaso, Christophe and Dubois, Emmanuel *Model Assisted Creativity Sessions for the Design of Mixed Interactive Systems: a Protocol Analysis*. (2013) In: 14th IFIP Technical Committee 13 on Human-Computer Interaction International Conference (INTERACT 2013), 2 September 2013 - 6 September 2013 (Cap Town, South Africa).
# Model Assisted Creativity Sessions for the Design of Mixed Interactive Systems: A Protocol Analysis

Christophe Bortolaso<fnref n="1" /> and Emmanuel Dubois<fnref n="2" />

**Abstract.** To help designers face the complexity of mixed interaction and identifying original and adapted solutions, we developed and evaluated an original approach to interaction design. This approach, called Model Assisted Creativity Sessions (MACS), aims to combine the best elements of both a model of mixed interaction, and a collaborative and creative session. The objective is twofold: to support the exploration of the design space, and to establish a common language between participants. To assess the viability of this approach, we relied on a protocol analysis of the verbal recordings of two existing design situations. Results show that the model impacts the generation of ideas and that participants use the model concepts to share their thoughts during the session.

**Keywords:** Design, Method, Mixed Interaction, Model, Creativity.

## 1 Introduction

In the last ten years, we have seen an incredible amount of new interactive systems combining physical and digital entities. The emergence of augmented reality in games such as *EyePet*<fnref n="1" />, the development of Kinect based interaction techniques such as *Omnitouch* [15], the commercialization of *Sifteos'* cubes<fnref n="2" />, etc. constitute relevant instances of the huge potential offered by the combination of physical and digital worlds. These interaction forms are now commonly known under several terms such as mixed interaction, augmented reality, tangible user interfaces or even pervasive systems. To refer to all these advanced forms of interaction, we will use in this paper the general term of Mixed Interactive Systems (MIS).

With regards to traditional interactive systems (i.e. WIMP) designing MIS implies *many additional considerations*. For instance, physicality and topological properties of the involved artifacts, mediums enabling the communication between physical and digital worlds, spatial and semantic relationships between physical and digital artifacts or the quasi-infinite possibilities in terms of input/output technologies are dimensions that designers do not need to consider when designing traditional interfaces.

<footnote n="1">¹ School of Computing, Queen's University, Kingston, Ontario, Canada K7L 3N6</footnote>

<footnote n="2">² IRIT - ELIPSE, University of Toulouse, 31000 Toulouse, France
Christophe.Bortolaso@queensu.ca,
Emmanuel.Dubois@irit.fr</footnote>

<footnote n="1">¹ EyePet™, http://www.eyepet.com/</footnote>

<footnote n="2">² Sifteo, https://www.sifteo.com/</footnote>
Due to these new forms of inputs, these systems are growing in complexity, and the amount of possibilities in terms of interaction techniques is exploding. Indeed, during the development of a MIS, designers are required to manage not only the general complexity of HCI, but also all these specific considerations. The design space is therefore broader, richer and harder to explore. This implies many conceptual and *methodological problems* [26] and resulting designs often stay focused on archetypical solutions.

Based on this assessment, we wondered which support would help designers to better face, explore and take advantage of this complex design space. How do we help designers to look past these trivial and mostly technology driven design solutions? Providing such a support would also contribute to the diversification of MIS by helping designers identify original and adapted solutions.

Our contribution in this paper consists in revisiting the ideation step in the design process of MIS. Interaction design being intrinsically interdisciplinary, we focus in this paper on the collaborative and explorative part of the design process. In the following sections, we first provide the necessary background related to explorative design. We then elaborate on existing design approaches and resources in the domain of HCI and mixed interaction. We then introduce and illustrate a design method called MACS, which stands for Model Assisted Creativity Session. We continue by showing the viability of this approach through the analysis of the use of MACS in two real design situations. We finally conclude on the benefits of this approach and the perspectives it offers in terms of design and research.

## 2 Toward a New Form of Creative Design

First, as already pointed out by Visser [30], the design activity can be considered as a *construction of representations*. Through this paradigm, design has to be viewed as an iterative task involving the externalization of ideas, the creation of representative artifacts, and the reinterpretation of the produced representations. Through this process designers infer properties of the design that were not perceptible before. It ultimately leads to new insights that contribute to the discovery of a satisficing solution [27]. This process, widely described by Schön [25] as *reflection-in-action*, constitutes the basics of any design activity.

Furthermore, it is now commonly accepted that the design process can also be understood as the exploration of a space whose limits are blurry and ill-defined [27]. As defined by Gero [12], this exploration takes place in a space where variables (i.e. properties of the designed artifact) are adjusted. This space can be subdivided into three different sub-spaces [12]:

* The *routine design space* defined as a space of potential designs, in which all the variables and their applicable ranges of values are directly extracted from existing solutions.

* The *innovative design space* defined as a space of potential designs in which all the instantiated variables are known but which values are new or unfamiliar.

* The *creative design space* defined as a space that uses new variables. This generally leads to a paradigm switch.
Accessing the Innovative and Creative space requires the knowledge of variables (see Fig. 1). For example, when designing a MIS, considering both semantic relationships between physical and digital artifacts, necessitates more intelligible designs.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Access to the Innovative space through the manipulation of variables</figcaption></figure>

In the field of HCI, a set of widely accepted representations is used as a support to the instantiation of variables related to interaction. Scenarios, sketches, storyboards, mock-ups, etc. have proven to be efficient and flexible representations. In addition, these informal representations have the huge advantage of being understandable by most of the stakeholders in the design process such as end-users. Therefore these representations play the role of boundary objects that allows “*people from different areas of expertise to bridge their separate knowledge domains, to create a shared understanding, and improve decision making*” [7].

However, their contribution to the exploration of the design space is limited to the manipulation of the variables that the designer is already familiar with. As a result, the expressive and generative power of these representations highly depends on the knowledge and level of skillfulness of the designer [31] itself: the access to the Innovative and Creative spaces are then highly influenced by the designer's experience. The solution to this limitation could rely on the use of a specific language. Indeed, as stated by Löwgren and Stolterman [22], in interaction design “*a language is necessary in making ideas and thoughts more precise and well-crafted*”.

While many design methods focus on the use of traditional artifacts, we chose to explore how abstract concepts, theories and models can assist the design activity. Our research can therefore be seen as an instance of modern design theories such as C-K design theory [16] (i.e. in which the design activity is understood as an interplay between existing Concepts and domain Knowledge) or even as the stream recently highlighted by Stolterman and Wiberg in [28] and so-called *Concept-Driven Interaction Design*. These attempts result in a set of new methods, in which design takes inspiration and restriction from existing paradigms, theories and languages. To our knowledge, only a few HCI methodologies have tried to implement this approach. We synthesize a set of existing methods adopting this approach in the next sections.
# 3 Concept-Driven Interaction Design in Practice

In the field of HCI, the literature mentions a set of design methods that can be considered as Concept-Driven approaches. They mostly differ in the level of abstraction of involved representations.

First, some of them are focused on the use of concrete representations. One of the first methods implementing this paradigm is the PICTIVE method [24]. Proposed by Muller in 1991, it consists in asking end-users to create the interface of their dreams through the manipulation of a paper-based and predefined set of buttons, windows, lists, icons, etc. The proposed set of widgets is limited to the existing ones (e.g. buttons, lists, text fields, etc.), which therefore constrains the exploration in the innovative design space. However, PICTIVE remains dedicated to the design of WIMP interfaces. More recently, Koleva proposed a similar approach [21] but in order to explore the possibilities in terms of augmented reality for museum visits. In this experience, curators and engineers collaboratively manipulate a diagram representing component-based software assembly. The manipulated software components are instantiated on the fly from an existing list of available components. This list therefore represents a set of existing possibilities in terms of augmented reality solutions, but the creation of links between the different components remains the responsibility of the curators and engineers. However, approaches like PICTIVE or Koleva's, are mainly technologically driven, and the exploration consequently takes place in a space that is too concrete and narrowed.

To overcome such limitations, a set of approaches suggests a concept-based exploration at a higher level of abstraction. For example, the Rich Picture Method [23] suggests that a facilitator and several participants collaborate to draw a picture depicting stakeholders, relationships and their concerns. The Rich Picture notation provides all the concepts (i.e. the variables of the design space) that need to be expressed and link them in a diagrammatic form. Here the design space is not explored through a set of concrete interface elements, but at a higher conceptual level representing the concepts of the application domain. Finally, in the domain of Tangible User Interfaces (TUIs), Hornecker proposed an original method called the Card Brainstorming Game [17]. This game mainly consists of a brainstorming session in which participants are stimulated by provocative cards. Each card is related to a dimension expressed in the Tangible Interaction Framework [18], a conceptual paradigm of TUIs. As observed by the author, "it thus ensures that all the central tenets have been taken into account" at the end of the session. However, this last approach remains poorly generic and does not provide any systematic strategy to explore the design space.

From our point of view, these Concept-Driven Interaction Design instances converge on two aspects: 1) the manipulation of a defined and limited number of concepts and 2) the collaborative elaboration of a dedicated representation. This sort of abstract based exploration constitutes a very promising approach to interaction design and needs to be further investigated, evaluated and used in real design situations.
# 4 Formalism and Creative Design: A Promising Combination

The challenge of our research is to gain a better understanding of the benefits of exploring the design space through the use of a model of MIS. During the last five years, we tested and refined an original concept-driven method dedicated to the design of MIS. The approach we developed is called *Model Assisted Creativity Session (MACS)*. The goal of a MACS is to identify and describe a set of alternative solutions to a design problem. Such creative sessions usually involve between 5 and 7 participants including a facilitator. The core principle of MACS is to collaboratively take advantage of the concepts expressed and characterized in an existing model of mixed interaction. The participants thus generate ideas and encode them on the fly in the model's notation. For now, this method has been successfully used [6] with two different interaction models: ASUR [11] and MIM [9]. However, the literature mentioned a lot of other models that are candidates for use in a MACS: each one is describing different aspects of MIS and adopting a different paradigm, using different concepts and represented through various notations [18, 26, 29].

Beyond its role of providing a support to the description of generated solutions, the model is also intended to play a major role in the creative and collaborative process:

1. *Support the exploration of the design space through the manipulation of a limited number of concepts:* it enables participants to primarily focus on relevant aspects of the system to design. Limiting the participants to the exploration and recombination of these relevant aspects, thus stimulates the creativity of the participants by arousing unexpected combinations. Indeed, it is now common knowledge that a design activity always takes place in a constrained cognitive space [4] and that creativity highly benefits from these constraints [8].

2. *Support the co-construction of an external representation:* the model allows us to represent the generated solutions and creates a common language between disciplines. It thus contributes to the elaboration of a shared understanding of the problem. This phenomenon, known as cognitive synchronization, has proven to be one of the main criteria to the success of collaborative design activities [10].

These two objectives will constitute the basis of our further analysis and evaluation that we present in the next sections.

## 4.1 The MACS Method Definition

To reach the objectives that we defined in the previous section, the MACS method is composed of a set of steps and principles that the group should follow. More details about the core principle of this method are provided in [6] and examples of its use in different context have been illustrated in [5, 14]. However, to ease the understanding of this paper, we briefly describe the five main steps of the approach below:

**Step 1.** The facilitator introduces the model and its notation to all the participants. This introduction should not take longer than 15 minutes. The presentation of the concepts that will be manipulated and their associated representations contributes to
the cognitive synchronization between participants of different disciplines and enables
the use of the model. The facilitator needs therefore to be an expert of the model. At
the end of this step, we recommend to provide the participants with a concept remind-
er such as a legend of the model.

**Step 2.** The facilitator introduces the design situation and the problem to solve. As for step 1, this step should take no longer than 15 minutes. Any design artifact that describes a part of the design problem can be used at this point (e.g. mock-ups, videos, sketches, requirements list, etc.).

**Step 3.** This is the main step of the method in which participants will explore the design space. In practice, participants generate ideas to answer the interaction design problem and encode them on-the-fly into the model notation. The ideas take the form of model-concepts and/or links between them. The model notation also enables the group to keep track of the generated ideas and to iterate around the produced solutions. The facilitator must manage the group dynamics and can also help participants with the model's syntax.

**Step 4.** This step is conducted by the facilitator after the session and aims at extracting the best solutions from the idea generation. To do so, the facilitator characterizes the produced models with regards to the initial requirements composing the design problem. The facilitator therefore has to select solutions that best fit the requirements. He can also alternatively combine (part of) different solutions, each of them partially covering the requirements, in order to obtain complete solutions satisfying the requirements. At the end of this step, the best solutions are candidate to prototyping and reintroduced into the design process.

**Step 5.** As for step 4, the facilitator also conducts this step after the session. It is aimed at identifying the characteristics of the generated solution that need further refinement. This can happen if some of the model dimensions have not been discussed in step 3 or if the model does not express certain characteristics. In the first case it may lead to a new MACS using the same model or a different one more appropriate for the identified problem. In the second case it will require the use of another model or dedicated representation. In both cases, these issues open the design to further iterations.

This set of five steps constitutes the basis to conduct a session. To complete the definition of the method, the following section illustrates how the manipulation of the model performed during step 3 can contribute to the exploration of the design space.

## 4.2 How Can a Model Help to Generate Interactive Solutions?

In this section, we show how to generate systematically alternatives by using an interaction model. To illustrate this method, we chose a famous TUI: the Reactable [20]. This system allows users to compose electronic music by manipulating little bricks on an interactive tabletop. Each brick represents either a sample of music or sound effects that can be applied to it.
To demonstrate the power this approach, we will rely on a model of MIS called ASUR [11], which stands for Adapter, System, User and Real objects. ASUR defines a mixed interactive situation as a set of entities linked by information flows. More details about the ASUR model are available in [11].

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** 1) ASUR model of the task “adding a sample” with the Reactable; 2) Model 1 after materializing the digital entity *Music*, 3) Model 2 after removing the physical entity *Surface*, 4) Model 3 after grouping of *Music* and *puckSample* physical artifact</figcaption></figure>

For example, Fig. 2 n°1 (top-left corner model) shows the ASUR description of the Reactable system. This model describes the task “Adding a sample on the table”. It shows a User handling a physical entity called puckSample whose form and position are recognized by an adapter IRCamera, which is physically associated with a physical entity Surface. The adapter IRCamera notifies a digital concept called Sample, which acts on the main digital concept Music. Both Music and Sample digital concepts are returned to the User through the adapters Projector and Speakers, which respectively enable the visual feedbacks on the surface and the diffusion of the sound.

To generate alternative designs to the Reactable, we propose to modify consecutively three times the original model (Fig. 2 Model n°1) describing the existing Reactable system.

To begin, we propose to materialize the digital entity *Music* shown on model n°1 in Fig. 2. This results in creating a physical object, whose specific characteristics will change in function of the variation of the sound. The model n°2 of Fig. 2 shows the changes it impacts on the model. This solution leads to a new system in which the user sees/touches/feels physical variations in the music.

On this new basis, we suggest to modify this second model by removing an entity. To highly impact the system, we removed one of the most prominent objects in the interaction: the *Surface*. This leads to the model n°3 shown on Fig. 2. Here, the user can no longer put the puck on the surface and has to manipulate it directly. As a
result, several possibilities are now opened. For example, this puck could take the form of a cube whose position and orientation in the space act on the Music.

Finally, we can go even further by performing a third modification and grouping
several entities together. Based on model n°3, we can group the physical puck Sample and the materialized Music, the two objects we have created previously. It results in a fourth solution (see model n°4 on Fig. 2) in which the user manipulates an
object whose characteristics vary in function of the sound. For example, it could take
form of a malleable object whose form varies with the frequency of the Music and
which user can move, shake, and orient to act on the system. This is obviously an
imaginary solution, but tomorrow's materials could easily drive us to such interactive
systems.

To conclude, we have shown that by applying basic modifications on the original modeled solution, it is possible to explore the design space and discover unexpected solutions. The techniques we used to modify the model (i.e. materialization, removing and grouping) are applicable infinitely, in any order and could be completed by symmetrical techniques (i.e. dematerialization, adding, splitting).

This list of technique is definitely non-exhaustive, but provides a good overview of
how simple manipulations of a model can help in the finding of alternatives. Further
work needs to be conducted to identify a larger set of generative techniques and to
study their completeness.

In addition, these techniques should not be seen as an automatic way to generate
new solutions. Indeed, each time an entity is impacted by the use of a technique, we
replaced it by a new entity with an undefined semantic (cf. each “?” on the entities of
models 2, 3, 4 on Fig. 2). This means that the use of such manipulation is intended to
force designers to reinterpret the situation and to reconsider the initial problem. Such
manipulations do not generate ready-to-use solutions. These techniques therefore
need to be combined with traditional creative techniques such as divergent or analog-
ous thinking. When used in combination with the ASUR model, the MACS method
supports the combination of creativity and modeling, thus taking advantage of both.

**5 Analysis**

We showed that the use of a model during collaborative and creative session could
constitute a promising and alternative way of designing MIS. As underlined in the
introduction, the aim of such an approach is twofold.

First, the model is supposed to assist the exploration of MIS design space through
the manipulation of a limited number of concepts. While traditional representations
such as scenarios, sketches or mock-ups are known to impact collaborative and crea-
tive thinking, this remains uncertain with representations such as an interaction mod-
els. Therefore, we came up with the following hypothesis:

*Hypothesis 1: In a MACS, the model impacts the idea generation, i.e. the exploration of the design space.* To assess if the model has an impact on the generation of idea, we first need to verify if the model is used by the participants to support the
creative process. To do so, our following analysis will quantify the links between the
generation of ideas during the session and the use of model concepts during a session.

Secondly, the model is aimed at providing a language to the group that collaboratively
manipulates the concepts. Computer science engineers, designers, researchers and
end-users will use the words and the representations associated to the introduced con-
cepts to express their ideas during the sessions. Therefore, the model should establish
a common ground between participants from different backgrounds and disciplines.
Consequently, we hypothesized about this aspect as follows:

*Hypothesis 2: In a MACS, the model acts as a language of reference, i.e. support the collaboration during the session.* To assess this second hypothesis, we need to verify if the model’s concepts are actually used to express and discuss ideas during the sessions. To do so, our following analysis will focus on the use of model concepts during the session.

To verify these hypotheses, we performed a verbal protocol analysis of two case
studies that we detail hereafter.

**5.1 Case Studies**

Over the last five years we used the MACS method eight times with different case
studies, application domains, design teams and type of problems. Among these sever-
al applications, we selected two occurrences hereafter referred to as MACS<sub>a</sub> and
MACS<sub>b</sub> to be candidates for a detailed analysis. These two instances were concerned
with a project focusing on the development of MIS dedicated to cultural centers and
involved highly motivated interdisciplinary partners (computer scientists, ergonom-
ists, curators, dancers, etc.). These authentic design situations raised design questions
related to existing and concrete problems. Participants of these two instances of
MACS were thus intrinsically motivated by a clear and unique objective, which con-
stitutes a major aspect in the creative process [2]. Prior to these two sessions, partici-
pants (except the facilitator) had no knowledge about the ASUR model.

**MACSa: An Interactive Exhibit Dedicated to the Understanding of Cladistics.** In this session, participants were asked to identify input interactive techniques to manipulate a 3D tree representing a classification of species. The application, called CladiBubble, aims at explaining why some well-known groups of species, such as fishes or reptiles, are no longer valid in terms of cladistics. Cladistics is a scientific method that classifies species regarding to their phylogenetic criteria and is the main focus of the Museum of Toulouse's exhibits. With CladiBubble, Visitors are invited to manipulate a 3D cladogram in order to bring species spatially close to each other. They then bring them into a bubble to create a group. Finally, the application validates the group in terms of cladistics and provides cues to constitute a group that is correct.

At the beginning of the MACS, during step 2, we used a video prototype to dem-
onstrate the application. The design problem was to find mixed interaction techniques
to allow the manipulation of the cladogram and grouping of species with the bubble.
The design space was therefore opened to many input interaction techniques and ways of rendering the application in the museum. The session included five participants: a facilitator, a technical engineer, a HCI specialist, an ergonomist and a MIS specialist. The models were edited on a computer assisted graphical editor and projected on a wall that was visible by all the participants (see Fig. 3).

<figure><img src="image_6.png" /><figcaption>**Fig. 3.** The MACS<sub>a</sub> session. A participant is showing an element on the model, which is projected on the wall.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** The MACS<sub>b</sub> session. Participants are collaboratively editing a model with the paper based support.</figcaption></figure>

**MACSb: An Augmented Ballet to Perceive Dancer Emotions.** This session was related to the design of an augmented ballet. The main objective of this project was to emphasize the dancers’ emotions and to express it to the audience. To do so, a dancer’s sequences of moves were sensed through a motion capture suit. Each sequence of movement was then linked to a predefined emotion or set of emotions. The challenge of this MACS was to identify alternative ways of making the dancer's emotions perceptible to the audience. As a result, the design space was mainly open to output interaction techniques and representations of emotions.

The session included seven participants: a facilitator, an emotion specialist, a MIS specialist, a 3D specialist, a usability expert, a dancer and a final user (i.e. a person used to watching ballet dancing). Here, the models were edited with a tangible and collaborative editor (see Fig. 4) composed of large sheets of paper, pre-printed and colored post-it notes representing ASUR entities, and colored pens.

## 5.2 Protocol Analysis

To verify our two hypotheses and assess the viability of the MACS methods, we relied on the protocol analysis technique [13], a largely accepted methodology in the field of design research. This technique consists of coding a set of recordings of a defined task by following a specific scheme. In our case, we collected audio/video recordings of the two sessions and analyze them by using a specific scheme dedicated to the analysis of collaborative design sessions.

**Coding Scheme.** The coding scheme we used, is based on an existing method named COMET [10] which has been designed to analyze activities of designers during meetings. Here, we relied on the part of the method that is specifically dedicated to the analysis of dialogues. To do so, COMET sequentially relies on 1) the textual
transcription of the dialogues occurring in a design session, 2) the segmentation of each turn-taking into one or several segments conform to the COMET predicate format (MOD(ACT/ARG)) and 3) the coding of each segment.

<table><caption>**Table 1.** Our coding scheme. Gray lines represent the original solution code in the COMET [10] scheme. We divided it into 5 different codes in order to differentiate the different types of solutions that are relative to the model's concepts, or out of the model scope.</caption><thead><tr><th>CLASS</th><th>CODE</th><th>DEFINITION</th></tr></thead><tbody><tr><td rowspan="2">MOD</td><td>AS</td><td>Assertion</td></tr><tr><td>RE</td><td>Request</td></tr><tr><td rowspan="4">ACT</td><td>GE</td><td>Generate: Idea generation</td></tr><tr><td>EV</td><td>Evaluation: Judgment, criticizing</td></tr><tr><td>IT</td><td>Interpretation: Rephrasing, reformulation</td></tr><tr><td>IF</td><td>Information: Completing, adding, specifying</td></tr><tr><td rowspan="8">ARG</td><td>MS</td><td>Model Metaphor: General idea, a whole model</td></tr><tr><td>ME</td><td>Model Entity: Physical/digital/Adapter/User artifact</td></tr><tr><td>MA</td><td>Model Attribute: Characteristic of an entity</td></tr><tr><td>MC</td><td>Model Channel: Communication between entities</td></tr><tr><td>OT</td><td>Other Solution: Idea out of the model scope</td></tr><tr><td>PD</td><td>Problem Data: Part of the design problem</td></tr><tr><td>DO</td><td>Domain Objects: Concept of the application domain</td></tr><tr><td>DP</td><td>Domain Procedures: Constraint of the application domain</td></tr><tr><td>GO</td><td>Goals: main objective of the application</td></tr><tr><td>TA</td><td>Tasks: atomic task in the use of the application</td></tr></tbody></table>

This scheme is divided into three classes (see Table 1). The first class, called Modulation, is relative to the type of intervention (MOD=assertion | request). The second class, called Action, is relative to the type of action (ACT=idea generation | evaluation | information | interpretation). The third class, called Argument, is relative to the subject that is currently referenced (ARG=solution | problem data | domain object | domain procedure | goal | task). As a result, each segment of the protocol will be coded as being a Modulation, an Action and an Argument.

To further adapt the use of COMET to our context, we added a level of precision to the solution code in the argument class (see gray lines on Table 1). Initially in COMET, the solution code should be attributed to segments in which a participant elicits a solution answering the design problem. With our adaptation, solutions can be 1) relative to the model's concepts, or 2) out of the model scope. In order to analyze and understand the use of the model's concepts during the session, four possible values, representative of the concepts expressed in the model used, have been considered for the solution codes: metaphor, entity, attribute and channel. Table 1 above summarizes all the codes composing our final coding scheme.
**Corpus Features & Protocol Elaboration Method.** First, as shown on Table 2, both sessions combined, our analysis has been conducted on more than 2700 segments, which represent more than 26000 words.

<table><caption>**Table 2.** Corpus features and Kappa agreement coefficients between the coders and protocols</caption><thead><tr><th colspan="3">MACS<sub>a</sub></th><th colspan="3">MACS<sub>b</sub></th></tr><tr><td colspan="3">Duration = 1:40:46<br/>nbSeg = 974 segments (9896 words)</td><td colspan="3">Duration = 2:02:32<br/>nbSeg = 1815 segments (16861 words)</td></tr><tr><th>C1&amp;C2</th><th>C1&amp;AP</th><th>C2&amp;AP</th><th>C1&amp;C2</th><th>C1&amp;AP</th><th>C2&amp;AP</th></tr></thead><tbody><tr><td>K=0.59</td><td>K=0.61</td><td>K=0.71</td><td>K=0.69</td><td>K=0.70</td><td>K=0.77</td></tr></tbody></table>

C1&C2: Agreement between coder 1 and coder 2.

C1&AP: Agreement between coder 1 and arbitrated protocol.

C2&AP: Agreement between coder 2 and arbitrated protocol.

However, even with such a large sample, the protocol analysis method is known to be influenced by the interpretation of the individual that performs the coding. To reduce this interpretation effect and tend to a form of objectivity, we relied on a specific coding process, as described in [13]. Therefore, the protocols of both case studies have been coded by two different individuals and then arbitrated within more than ten days after the coding. We then computed a level of agreement between the different protocols based on Cohen' Kappa coefficient. Table 2 shows the Kappa coefficient we obtained. The coefficients between 0.61 and 0.8 indicate a substantial and good agreement between the coders. Given our context, the data can be considered as reliable. All the following analysis has been realized on the arbitrated protocol (AP).

# 6 Results

Both sessions have been conducted successfully and produced modeled solutions that answer the design problem.

MACS<sub>a</sub> has produced 17 models with various forms of MIS solutions. For example, participants have envisioned the use of a tangible pump to enlarge the bubble group in the game. Differently, another interesting solution was related to the use of several tangible bricks representing species that could be associated with a multi-function tangible selector. The display of the application has also been revisited through the use of an augmented shadow that allows visitors to point and select elements on the screen. However, due to the amount of ideas, we do not report all the modeled solutions here.

MACS<sub>b</sub> has produced 5 different models. During this session, the team grabbed the opportunity to reconsider several aspects of the project, which was ongoing for three years before this session. As a result, unanticipated solutions emerged. For example, a proposition was to relay all the data points, captured by a motion suits sensors, to the audience, rather than combining this information into a single representation. Participants therefore explored different representations and modalities such as numbers, colors, sounds, curves, etc. As a result, the emotion could be interpreted by the spectator himself. Another original solution suggested using the dancer's movements could
be used to sculpt a virtual 3D object. The resulting 3D object would be returned to the audience through different projectors or augmented reality glasses.

In the following paragraphs, we present the data we collected from the protocol analysis. Even if the differences between these two case studies make them not rigorously comparable, this double analysis aims at showing the general trends we observed in the use of the MACS method with the ASUR model.

## 6.1 Overall Observations on the Code Distribution

Before we focus on our hypotheses, we made a set of observations regarding the code distribution of our protocol analysis. First of all, as shown in Fig. 5, protocol analysis reveals that the code distribution is very similar over the two sessions. This encourages us to think that the method is stabilized and has predictable behavior, whatever the team, design problem or the support used to edit the model (PC based or paper based).

<figure><img src="image_8.png" /><figcaption>**Fig. 5.** Code distribution for each class (modulation, action, argument) and for both sessions</figcaption></figure>

However, we observed some meaningful differences between the two sessions. For example, the MACS<sub>b</sub> (ballet) shows a bigger interest in terms of model attributes (220 MA segments) and channel (248 MC segments) while MACS<sub>a</sub> (museum) remains mainly focused on the reference to model entities (328 ME segments). This difference is due to the design problem itself. Indeed, as elicited in the case study presentation, the question relative to ballet augmentation (MACS<sub>b</sub>) was more prone to the exploration of emotions rendering, which implies the use of attributes and channels in the model. In contrast, the interaction technique problem of CladiBubble (MACS<sub>a</sub>) was more focused on the identification of artifacts to be manipulated by the user.

Another interesting difference is relative to the reference to domain objects (DO). MACS<sub>b</sub> was highly more focused on objects relative to the ballet and to the dancing while participants of the MACS<sub>a</sub> made almost no reference to the domain objects (concepts relative to cladistics and museums). The level of definition of the design problem can explain this difference. Indeed, MACS<sub>a</sub> occurred while the CladiBubble prototype was in a real concrete state, while when MACS<sub>b</sub> occurred, the ballet augmentation project was essentially starting from scratch.
These two observations tend to show that 1) participants are able to use the dimensions of the model that are the most relevant to a specific design problem and 2) the MACS method is adapted for design problems with different levels of definition.

## 6.2 Hypothesis Assessment

**Hypothesis 1: The Model Impacts the Idea Generation.** To assess this first hypothesis, we focused on the solutions to the problem generated by the participants and on the actions of generation. As a result, we first looked at all the segments which argument was coded as a solution to the design problem (ARG=MS | ME | MA | MC | OT) and then at the links between action of generation (ACT=GE) and the references to a model concept (ARG=MS | ME | MA | MC).

On this basis, we computed the ratio R1 (1) of solution relating to a model concept on the total amount of argument related to a solution.

$$R1 = \frac{\sum_{i=0}^{nbSeg} \{MS,ME,MA,MC\}}{\sum_{i=0}^{nbSeg} \{MS,ME,MA,MC,OT\}} \quad (1)$$

R1=92% for of MACS<sub>a</sub> and R1=90% for MACS<sub>b</sub>. This means that when participants are talking about solutions to the design problem, the very large majority of their verbal intervention is referring to a model concept.

To go further in the dynamic of the session, we also analyzed the sequencings between actions of generation and argumentation about the model concepts. We therefore counted the number of times a participant referencing a model concept (ARG=MS | ME | MA | MC) was followed by a participant generating an idea (ACT=GE). Table 3 illustrates the kind of situation we considered. This analysis deeply relies on the situated point of view of design. Indeed, design can be seen as an activity that is sequentially and contextually anchored [10]. Through this point of view, a MACS is a constant exchange between participants themselves and between participants and the model. Each turn taking can therefore influence the next one.

<table><caption>**Table 3.** Example of considered sequences. In this sample, extracted from the MACS<sub>a</sub>, the Facilitator's segment 232 triggers the idea generation action in the Technician's segment 233.</caption><thead><tr><th>Seg. Time</th><th>Participant</th><th>Transcription</th><th>MOD</th><th>ACT</th><th>ARG</th><th>Solution</th></tr></thead><tbody><tr><td>231</td><td>0:58:05 Ergonomist</td><td>This is more precise than a camera!</td><td>AS</td><td>IT</td><td>ME</td><td></td></tr><tr><td>232</td><td>0:58:09 Facilitator</td><td>But what's the interest in using <u>accelerometers and gyros?</u></td><td>RE</td><td>IT</td><td>ME</td><td></td></tr><tr><td>233</td><td>0:58:25 Technician</td><td>This could be used for <u>gestures when you move the nodes?</u></td><td>AS</td><td>GE</td><td>MC</td><td>Model 3: Gestures to move nodes</td></tr><tr><td>234</td><td>0:58:30 Facilitator</td><td>Oh yes! We could use gestures!</td><td>AS</td><td>IT</td><td>MC</td><td></td></tr></tbody></table>

We observed that 64% of actions of generation were preceded by a reference to a model concept for the MACS<sub>a</sub> and 62% for MACS<sub>b</sub>. In other words, the majority of the idea generations occurred directly after a participant would mention an idea in the scope of the model's concepts. As a result, we conclude that the majority of the generative actions are stimulated by a model concept.
Given that the large majority of the topics discussed during both sessions are related to the model framework and that the model concepts stimulate the idea generation process, we validate our first hypothesis.

**Hypothesis 2: The Model Acts as a Language of Reference.** To assess this second hypothesis, we focused our analysis on amount of references to the model concepts without considering the type of action (ACT=GE | EV | IT | IF) and on salient behaviors in the video recording of the sessions.

<figure><img src="image_9.png" /><figcaption>**Fig. 6.** An example of the collaborative use of the model</figcaption></figure>

First of all, we observed from the protocol analysis of both sessions that all the participants make reference to the model. In addition, we observed in the MACSb (in which a paper-based collaborative editor was used) that all the participants have contributed to the edition of the models. Finally, the model is also regularly used as a support to the referencing of already generated elements. For example, as shown on Fig. 6, participants point directly at an element on the model to support their assertions.

We then computed the ratio R2 (2) of references to the model concepts on the total amount of segments during the two sessions.

$$R2 = \frac{\sum_{i=0}^{nbSeg} \{MS, ME, MA, MC\}}{\sum_{i=0}^{nbSeg} \{MS, ME, MA, MC, OT, PD, DO, DP, GO, TA\}} \quad (2)$$

R2=68% for MACS<sub>a</sub> and R2=55% for MACS<sub>b</sub>. This means that for both sessions the majority of segments were related to a model concept (ARG=MS | ME | MA | MC). This also means that the discussion is mainly driven by the use of the model to generate, evaluate, interpret and inform ideas.

We also considered the temporal aspect of the sessions. We computed the average occurrence frequency of a reference to a model concept. It appears that over the total duration of the session, a reference to the model occurs each 9.11 seconds (SD=16.78) for the MACS<sub>a</sub> and each 8.75 seconds (SD=26.38) for the MACS<sub>b</sub>. Even if the standard deviation is high in regard to the time, we can conclude that references to the models concepts occur very frequently during the sessions.
These set of qualitative observations and computations lead us to conclude that the model effectively plays the role of the language of reference during the sessions, which consequently validates our second hypothesis.

# 7 Discussion and Perspectives

In this paper, we first summarized the principles of a design method combining the best elements of both a model of mixed interaction, and a collaborative and creative session. We then studied the viability of this approach in order to understand if a design team can benefit from a formal representation to collaboratively explore the design space of mixed interaction. Regarding to previous evaluations of this method which were mainly based on qualitative results and informal observations [6], we demonstrated here the viability of the MACS method through a quantitative analysis based on verbal protocols. The results presented here show that in a MACS, the model impacts the generation of ideas (H1) and acts as a language of reference (H2).

Participants also perceived these two advantages. Indeed, during the sessions' debriefings, participants reported some fruitful comments. For example, at the end of the MACS<sub>b</sub>, the ballet dancer argued that during the session “*the model really allows to turn around an idea to find variation or combinations*”. A few minutes later he added: “...suddenly, I feel I can speak at levels in which I usually do not say anything, because I know nothing about it!”. During the same debriefing, the emotion specialist pointed out that “*the model has provided a common basis for support the communication between computer scientists and others*”. These interventions clearly indicate that participants felt the help provided by the model to explore the design space and to support the communication between people from different backgrounds.

However, to strengthen the value of our approach, it would be interesting to quantify to which extent the introduction of the model impacts the idea generation. To do so, it is required to compare the MACS with other creative methods such as a brainstorming. This could allow us to quantify the gain of the MACS in terms of design space exploration or the impact of the model manipulations on participants' cognitive load for example. Seeing the results we obtained to show the viability of our approach in this paper, we think that the MACS method is now mature enough to perform such a comparison. Techniques like linkography constitute relevant analytic tools to perform such a comparison.

In addition, the analysis we presented here could be extended to a behavioral analysis. In fact, the analysis of participants' physical and perceptual actions could 1) contribute to a better understanding of the model's impact during these sessions and 2) provide clues to the development of interactive tool dedicated to creative groups.

It is also important to underline that we learned from our experience the MACS approach is compatible with traditional HCI development process methods. In fact, during the development of the two case-studies presented here, we successfully used the MACS approach in parallel to other traditional participatory design techniques. In fact, the solutions modeled during the session can be used as a starting point for subsequent design steps, in particular when it comes to prototyping and implementation.
Indeed, model transformations are useful tools to infer (parts of) the running prototype from a source model such as those used in a MACS.

In addition, a MACS is supposed to work with any model. As a result, we could envision the use of other frameworks such as RBI [19] or TAC [29]. However, several reasons drove our choice to the use of ASUR. First, ASUR model is composed of a small number of concepts. Participants can thus roughly represent an interaction technique by manipulating less than seven types of entities, which make the model easy to handle and to learn. Secondly, ASUR is supported by a graphical representation. Several participants reported at the end of sessions, that they used the diagram's form to remember and to come back on some generated ideas. It therefore appears that the model's number of concepts and notation play an important role in the success of these sessions. An analysis based on the cognitive dimensions of notation [3] could be helpful to drive the choice of a model. For example, the viscosity and the level of abstraction of the notation appear to be relevant dimensions to focus on.

To conclude, we strongly believe that concept driven interaction design approaches are promising approaches in the field of HCI and more specifically for the design of advanced interactive systems. Indeed, these types of approaches have proven to be efficient in more established disciplines. For example, in the domain of physics, engineers can rely on a 40 dimensions matrix called TRIZ [1], to explore the design space and find original solutions to problems. This approach has proven to be efficient and is now used by many practitioners in the domain of physics all over the world. The research we conducted here is thus a step into the development of such a systematic approach dedicated to mixed interaction design.

## References

1. Altshuller, G.: Innovation Algorithm: TRIZ, systematic innovation and technical creativity. Technical Innovation Ctr. Retrieved from, p. 312 (1999)

2. Amabile, T.M.: Creativity in context. Boulder, p. xviii, 317. Westview Press (1996)

3. Blackwell, A., Green, T.R.G.: Notational Systems - the Cognitive Dimensions of Notations framework. In: John, M.C. (ed.) HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science (2002)

4. Bonnardel, N.: Towards understanding and supporting creativity in design: analogies in a constrained cognitive environment. Knowledge-Based Systems 13(7-8), 505-513 (2000)

5. Bortolaso, C., Bach, C., Dubois, E.: MACS: combination of a formal mixed interaction model with an informal creative session. In: EICS 2011, Pisa, Italy, pp. 63-72. ACM, New York (2011a)

6. Bortolaso, C., Bach, C., Dubois, E.: Co-Design of Interactive Museographic Exhibits: the MIME case study. In: ReThinking Technology in Museums, pp. 37-48. University of Limerick, Limerick (2011b)

7. Bresciani, S., Blackwell, A., Eppler, M.: A Collaborative Dimensions Framework: Understanding the Mediating Role of Conceptual Visualizations in Collaborative Knowledge Work. In: Hawaii International Conference on System Sciences, p. 364. IEEE Computer Society, Los Alamitos (2008)

8. Costello, F.J., Keane, M.T.: Efficient Creativity: Constraint-Guided Conceptual Combination. Cognitive Science 24, 299-349 (2000)

9. Coutrix, C., Nigay, L.: Balancing physical and digital properties in mixed objects. In: AVI 2008, Napoli, Italy, pp. 305-308. ACM, NY (2008)
10. Darses, F., Détienne, F., Falzon, P., Visser, W.: A Method for Analysing Collective Design Processes. In: ECCE 2010, Linköping, Sweden (2010)

11. Dubois, E., Gray, P.: A Design-Oriented Information-Flow Refinement of the ASUR Interaction Model. In: Gulliksen, J., Harning, M.B., van der Veer, G.C., Wesson, J. (eds.) EIS 2007. LNCS, vol. 4940, pp. 465–482. Springer, Heidelberg (2008)

12. Gero, J.S.: Design prototypes: a knowledge representation schema for design. AI Magazine 11(4), 26 (1990)

13. Gero, J.S., McNeill, T.: An approach to the analysis of design protocols. Design Studies 19(1), 21–61 (1998)

14. Graham, N., Dubois, E., Bortolaso, C., Wolfe, C.: Scenarchitectures: The Use of Domain-Specific Architectures to Bridge Design and Implementation. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part II. LNCS, vol. 6947, pp. 341–358. Springer, Heidelberg (2011)

15. Harrison, C., Benko, H., Wilson, A.D.: OmniTouch: wearable multitouch interaction everywhere. In: UIST 2011, p. 441. ACM Press, New York (2011)

16. Hatchuel, A., Weil, B.: C-K design theory: an advanced formulation. Research in Engineering Design 19(4), 181–192 (2008)

17. Hornecker, E.: Creative idea exploration within the structure of a guiding framework: the card brainstorming game. In: TEI 2010, pp. 101–108. ACM Press (2010)

18. Hornecker, E., Buur, J.: Getting a grip on tangible interaction: a framework on physical space and social interaction. In: CHI 2006, pp. 437–446. ACM, Montréal (2006)

19. Jacob, R.J.K., Girouard, A., Hirshfield, L.M., Horn, M.S., Shaer, O., Solovey, E.T., Ziegelbaum, J.: Reality-based interaction: a framework for post-WIMP interfaces. In: CHI 2008, p. 201. ACM Press (2008)

20. Jordà, S., Geiger, G., Alonso, M., Kaltenbrunner, M.: The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. In: TEI 2007, pp. 139–146. ACM (2007)

21. Koleva, B., Egglestone, S.R., Schnädelbach, H., Glover, K., Greenhalgh, C., Rodden, T., Dade-Robertson, M.: Supporting the creation of hybrid museum experiences. In: CHI 2009, p. 1973. ACM Press, New York (2009)

22. Löwgren, J., Stolterman, E.: Thoughtful interaction design. MIT Press, Cambridge (2004)

23. Monk, A., Howard, S.: Methods & tools: the rich picture: a tool for reasoning about work context. Interactions 5(2), 21–30 (1998)

24. Muller, M.J.: PICTIVE - an exploration in participatory design. In: CHI 1991, pp. 225–231. ACM, New Orleans (1991)

25. Schön, D.A.: The Reflective Practitioner: How Professionals Think In Action, 1st edn., pp. 305–308. Basic Books (2008)

26. Shaer, O., Jacob, R.J.K.: A specification paradigm for the design and implementation of tangible user interfaces. TOCHI 16(4), 1–39 (2009)

27. Simon, H.A.: Les Sciences de l'Artificiel, 3ème edn., p. 215. Folio (1969)

28. Stolterman, E., Wiberg, M.: Concept-Driven Interaction Design Research. Human-Computer Interaction 25(2), 95–118 (2010)

29. Ullmer, B., Ishii, H., Jacob, R.J.K.: Token+constraint systems for tangible interaction with digital information. TOCHI 12(1), 81–118 (2005)

30. Visser, W.: Designing as construction of representations: a dynamic viewpoint in cognitive design research. Human Computer Interaction 21(1), 103–152 (2006)

31. Youn-kyung, L., Sang-Su, L., Da-jung, K., Lim, Y., Lee, S.-S., Kim, D.: Interactivity Attributes for Expression-oriented Interaction Design. IJDesign 5(3), 113–128 (2011)