# Activity Theory as a Tool for Identifying Design Patterns in Cross-Modal Collaborative Interaction

Oussama Metatla, Nick Bryan-Kinns, Tony Stockman, Fiore Martin

<figure><img src="image_1.png" /></figure>

## To cite this version:

Oussama Metatla, Nick Bryan-Kinns, Tony Stockman, Fiore Martin. Activity Theory as a Tool for Identifying Design Patterns in Cross-Modal Collaborative Interaction. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. pp.232-240, 10.1007/978-3-642-40477-1_14 . hal-01504886

HAL Id: hal-01504886

https://inria.hal.science/hal-01504886

Submitted on 10 Apr 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Activity Theory as a Tool for Identifying Design Patterns in Cross-modal Collaborative Interaction

Oussama Metatla, Nick Bryan-Kinns, Tony Stockman, Fiore Martin

School of Electornic Engineering & Computer Science, Queen Mary University of London.

{oussama, nickbk, tonys, fiore}@eecs.qmul.ac.uk

**Abstract.** This paper examines the question of how to uncover patterns from the process of designing cross-modal collaborative systems. We describe how we use activity patterns as an approach to guide this process and discuss its potential as a practical method for developing design patterns.

## 1 Introduction

A key challenge in the design of interactive systems is how to leverage existing design knowledge. Design patterns play a role in addressing this challenge by capturing and communicating reusable design knowledge on how to map requirements to design solutions. In the field of human-computer interaction (HCI), design patterns have received increasing attention and sets of patterns have been published for a variety of application domains (e.g. [4, 14]). Similarly, a number of studies within the area of computer-supported cooperative work (CSCW) have addressed the question of how to use design patterns to document successful design knowledge, some of which have generated patterns that can be readily employed to develop practical solutions [2].

We are interested in the design of interactive systems that support collaboration between individuals who use different sets of modalities to interact with each other. Such differences can arise because of permanent sensory impairments or due to the capabilities of the technology used by each collaborator and the context of their interactions with it; we refer to group interactions in these settings as *cross-modal collaboration*. In this paper, we focus on exploring the process by which design patterns for cross-modal collaboration can be identified. We describe how we used an approach based on activity patterns [10] to uncover design patterns from two phases of a typical design process. First, we exemplify the application of activity patterns by reflecting on the evaluation phase of a cross-modal tool that supports collaboration between visually impaired and sighted coworkers. Second, we outline how activity patterns can be used to capture insights from a participatory design workshop that we conducted with visually impaired audio producers and musicians in order to understand their work process and gather requirements about how to design technology that could contribute towards increasing their inclusion in their workplace.
## 2 Cross-modal Collaborative Interaction

Despite significant progress in the use of audio and haptics in multimodal interaction design, research into cross-modal collaboration has so far remained sparse. Initial investigations have nonetheless identified a number of issues that impact the efficiency of collaboration in cross-modal settings. For example, an examination of collaboration between sighted and visually impaired individuals on an interactive puzzle game highlighted the importance of providing visually impaired collaborators with a continuous display of the status of the shared game [15]. Providing collaborators with independent views of the shared space, rather than shared cursor control, was also found to improve orientation, engagement and coordination in shared tasks. In another study [12], a multimodal system combining haptic devices with speech and non-speech auditory output was used to examine collaboration between pairs of visually impaired users on graph reading tasks. Results showed that the use of haptic mechanisms for monitoring activities and shared audio output improves communication and promotes collaboration. Although scarce, the literature on cross-modal collaboration has therefore started to generate insights into the knowledge that is needed to come up with effective designs to support interactions involving individuals with differing perceptual abilities across various domains. We propose to use design patterns as a means to capture such knowledge so that it can be effectively leveraged.

## 3 Design Patterns

The concept of design patterns originated from the field of urban architecture as a systematic approach to capturing solutions to architectural design problems [1]. Each pattern would characterise a given problem in terms of its context, a set of conflicting forces that are at play, a solution to the problem and the consequences of its application. Particular attention is given to the presentation of patterns as a clear and generic set of instructions so that they can be applied in new situations. A critical aspect of design patterns is that they are cross-referenced. Each pattern refers to higher level patterns describing further context and lower level patterns that could be used to refine the solution. This hierarchical organization structures a collection of patterns into a meaningful network and results in what is called a *pattern language* [7]. The notion of pattern language has also been used to emphasise that patterns should support a "lingua franca" amongst all the parties involved in a design process [9]. Patterns in this case can be used as a common vocabulary to improve communication of design expertise between stakeholders [4]. Indeed, design patterns have been shown to have a considerable potential as a method to support participatory design activities [6].

## 4 Uncovering Design Patterns

In both software engineering and HCI, design patterns are often derived from a reference to existing design solutions and hence are the product of observation, trial and
error, and experience [5, 7]. In CSCW, finding patterns has also been described as a
process that relies on extensive observation of how people use tools in context. For
instance, [11] followed a retrospective approach to examine data from field studies of
cooperative interaction in order to uncover patterns that could be generalised across a
number of work environments.

The body of work on patterns in CSCW does not always make it clear how patterns
come into existence in the first place, however. So the question often remains whether
the Alexandrian way of identifying and organising patterns should be used to guide
this process, or whether a different approach should be adopted. On a closer look, the
hierarchical nature of the scales of patterns described in [1] does not match the hierar-
chy of problems that are typically encountered in CSCW and an appropriation is
therefore required to understand the problem-hierarchy that is relevant to CSCW do-
mains. The domain of cross-modal collaborative systems design presents a set of chal-
lenges often characterised by the conflicts associated with reconciling interaction that
occurs across more than one modality [13]. In this case, we are interested in uncover-
ing patterns both from direct observation of use of cross-modal collaborative systems
as well as from design activities that seek to improve such systems.

## 4.1 Activity Patterns

We consider activity patterns [10] as a potential guiding framework for addressing the issues highlighted above. According to this framework, Alexander's patterns could be appropriated to embody the principals of activity theory (AT) and hence could be used to analyse activity in terms of understanding tool-mediated work in its context [3]. AT views human activity in terms of a system of tool-mediated actions carried out by a subject (i.e. and individual or a group of individuals) in order to achieve a desired outcome. Actions are characterised in terms of how they are organised within a community context, and how they are regulated by internal rules and mediated by a division of labour. This unit of analysis is conventionally represented by a triangular model to show how its elements interact with each other [8] (see Figure 1).

<figure><img src="image_3.png" /><figcaption>Fig. 1. An activity system as conceptualized by activity theory.</figcaption></figure>
According to [10], there are parallels between the design patterns principles introduced by Alexander and that of human activity as conceptualised by the unit of analysis in AT. These include:

* The definition of a pattern in terms of three related components expressing the relationship between a given context, a problem and its solution, which is consistent with the method of AT.

* The characterisation of a problem in a given context as being caused by a system of conflicting forces that arise in that context, which could be captured through AT's conceptual tool of *contradictions*.

Additionally, the hierarchical levels of activity in AT can also be used in a similar way to the Alexandrian concept of scales to help structure the scope covered by activity patterns. [1] used scales to organise their patterns at different levels of architectural design, e.g. city, road, building, room, window, etc. The concept of activity levels in AT are hierarchically structured into three levels, where a given *activity* is realised through a set of concrete *actions*, which are in turn accomplished through a series of *operations*. This gives patterns a sense of scale from high-level activities down to low-level operations. According to [10], patterns could be written to reflect each element in a given activity system – the design of *mediating artefacts*; the work of a *subject*; the *rules* and procedures; and the roles within the *division of labour* or *community* of the work group – as well as organised into a coherent pattern language that preserves the unity of these elements within each of the three levels of activity.

## 4.2 Activity Patterns in the Evaluation Phase

In order to assess the feasibility of this organisational framework in supporting the process of uncovering design patterns, we applied it to data gathered from two phases of a typical process of designing for cross-modal collaboration; evaluation and requirements capture. In the first instance, we used the activity patterns approach to reflect on the design of an existing cross-modal tool, which we evaluated with visually impaired and sighted users.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Sighted and visually impaired colleagues editing a shared diagram.</figcaption></figure>
The tool combines a visual diagram editor with auditory and haptic capabilities to allow simultaneous visual and non-visual interaction. That is, two coworkers collaborate on shared diagrams by accessing and editing them through the visual modality (for the sighted user) and the combination of audio and haptic modalities (for the visually impaired user). We deployed this collaborative tool in various workplaces including a local government office and a charity organisation where visually impaired and sighted coworkers access and edit diagrams as part of their daily jobs [13]. In the following, we describe two examples of applying the activity pattern approach retrospectively to analyse data that we gathered from the field studies and identify potential patterns for cross-modal design.

**Example 1: Consistency of interaction steps.** In this example, a visually impaired manager (VI) and their sighted assistant (S) at a local government office edit an organisation chart to reflect recent changes in managerial structures. At one point during the interaction, the pair decides to create a connection between two nodes on the chart diagram to highlight a relationship between an existing and a new position. They do this while discussing how the tool should be used to create this relationship.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Two activity systems for creating a connection between two nodes using the non-visual (1) and the visual (2) editors. Contradictions are highlighted with red circles and wavy arrows.</figcaption></figure>

To create a connection between two nodes using the non-visual audio-haptic editor, the visually impaired user must 1) browse the chart to locate the first node and select it, 2) browse the chart again to locate the second node and select it, 3) select the type of connection they wish to use and 4) issue a command to create the desired connection. To do the same in the graphical editor, the sighted user must 1) select the type of connection they wish to use from the graphical tool bar, 2) select the first node on the chart, 3) drag the connection towards the second node using the computer mouse, and
then 4) release the mouse to create the connection. Following the activity patterns approach, the actions of creating a connection between two nodes using the visual and non-visual editors can be represented as the two independent activity systems shown in Figure 3, which highlights a contradiction between the operational rules in the two activity systems; there is a mismatch between the interaction steps that each collaborator has to follow in order to create a connection between two nodes on the chart. Modelling the collaborative action of creating a connection in this manner has therefore uncovered a potential design flaw – which manifests itself as contradictions – that could hinder collaboration. Addressing this design flaw could lead to a design pattern that can eliminate the issues raised by the contradictions. For instance, the potential design pattern could describe the need to reconcile the two mediating artefacts in this context by ensuring equivalence and consistency of interaction steps between the visual and non-visual modalities.

**Activity patterns in the requirement phase.** While we used the activity patterns approach to drive retrospective analysis of data gathered from evaluation studies, we used it in a second instance to drive the design process from its initial requirement phase. In a second project, we conducted a participatory design workshop (PD) with visually impaired audio producers, engineers and musicians (Figure 5) in order to understand how they go about their work and what kind of difficulties they come across. Participatory design activities generate a huge amount of data and patterns could help with the process of organising the themes that emerge from this data. Here too, the concept of contradictions can be a useful guide to identify the tensions that exist in activity systems constructed to model the scenarios described by workshop participants, and hence could lead to insights about design solutions that could resolve such contradictions.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Participatory design workshop with visually impaired and audio producers.</figcaption></figure>

**Scenario 1: Exchanging formats between colleagues.** In a first scenario, while discussing his experience of working with sighted colleagues and clients, a visually impaired producer described his frustration with inaccessible plugins on digital audio workstations. The visually impaired producer explained how his work with sighted colleagues often involves exchanging projects back and forth in order to complete sub tasks involving the manipulation of audio captured using inaccessible formats or requiring interaction with inaccessible audio plugins. In some cases, these accessibility
issues have led to his exclusion from potential collaborative projects because the
standard formats used are not readily accessible or would take too long to work with.

**Scenario 2: Accessing a visual programming language.** In a second scenario, a visually impaired participant who specialises as an accessibility trainer described a similar experience with inaccessible visual tools. In this case, the issue was specific to working on projects that were coded using an audio programming language known as Max/MSP<fnref n="1" />, which is a visual programming language that uses diagrammatic representations as its main programming components. The visually impaired participant highlighted how inaccessible such programming languages are even though they are used to code audio, which could be considered a natural working modality for visually impaired individuals.

Both scenarios above could be captured by the activity system shown in Figure 6. Here, there is a clear contradiction between the subjects of the activity, i.e. the visually impaired audio producer and accessibility trainer, and the tools available to them as mediating artefacts in the context of their activities. Capturing these contradictions allows us to think about possible design solutions to eliminate them, which could eventually lead to the development of fully articulated design patterns that embody such solutions. For instance, the diagrammatic nature of the Max/MSP programming language could be made accessible using the potential design pattern described in Example 2 above, in which the spatial properties of a diagram are translated into hierarchical structures that support non-visual inspection and navigation.

<figure><img src="image_7.png" /><figcaption>**Fig. 5.** An activity system showing some contradictions captured through the PD workshop. Contradictions are highlighted with red circles and wavy arrows.</figcaption></figure>

We chose the above examples and scenarios to demonstrate how the potential design patterns they uncover could be structured around different hierarchical levels of activity. Each design pattern falls within one level of the hierarchy; example 1 was concerned with the operational rules for connecting two nodes on an organisational chart thus modelling activity at the *operational* level; and both scenarios 1 and 2 relate to a higher level *activity* involving the organisation of collaboration between

<footnote n="1">¹ See: http://cycling74.com/products/max/</footnote>
sighted and visually impaired stakeholders in sound production activities and the im-
pact of currently available tools on such organisation. Patterns can therefore be writ-
ten for different levels of activity and structured into a coherent pattern language,
which can facilitate both the creation of new patterns and navigation of a patterns'
space to help readers find their way through the design knowledge material.

# 5 Conclusion and Future Work

The design of cross-modal collaborative systems presents a unique set of challenges because such systems must allow individual users to equally contribute to the shared tasks while accommodating their individual perceptual differences. To date, no research has examined how to capture the knowledge required to design technology that makes cross-modal collaboration easier. Design patterns and pattern languages have a potential to facilitate the capture, presentation and communication of design knowledge. We proposed that designers and application domain experts in cross-modal collaboration could benefit from using design patterns as a uniform representation for expert knowledge. To this end, we explored the question of how potential patterns can be uncovered from an iterative design process and suggested that activity patterns could be used as a structured method to address this question. We also described how the activity patterns approach matches the domain in which individuals with differing perceptual abilities work together to achieve a common desired outcome through the mediation of a cross-modal tool.

One of the key benefits of using activity patterns to identify patterns is the concep-
tual tool of contradiction, which can be a useful guide for designers to identify the
tensions that exist in their designs when used in context and modelled as activity sys-
tems. We exemplified how this approach was useful for us in the evaluation phase of
a cross-modal collaborative tool as well as a technique for managing requirement data
from participatory design workshops. We plan to use this approach to both generate
and articulate an initial set of patterns to form a pattern language for designing cross-
modal collaboration, which we will then validate by applying the patterns in future
design iterations and incorporating them in future participatory design activities.

References

1. Alexander, C., Ishikawa, S., & Silverstein, M. (1977). A Pattern Language: Towns, Buildings, Construction (Center for Environmental Structure Series) Oxford University Press, New York.

2. Arvola, M., & Larsson, A. (2004). Regulating prominence: A design pattern for co-located collaboration. In *COOP 04, 6th Inter. Conf. on the Design of Cooperative Systems, 2004* (pp. 115-130). IOS Press.

3. Bertelsen, O. W., & Bødker, S. (2003). Activity theory. HCI models, theories, and frameworks: Toward a multidisciplinary science, 291-324.

4. Borchers, J. O. (2008). A pattern approach to interaction design. *Cognition, Communication and Interaction*, 114-131.

5. Coad, P. (1992). Object-oriented patterns. *Communications of the ACM*, 35(9), 152–159.
6. Dearden A, Finlay, J., Allgar, E., & McManus, B. (2002). Using pattern languages in participatory design. In Binder, T., Gregory, J., & Wagner, I. (Eds.), *Proc. of PDC 2002*. Palo Alto, CA.

7. Dearden, A., & Finlay, J. (2006). Pattern languages in HCI: A critical review. *Human-computer interaction, 21*(1), 49-102.

8. Engestrom, Y. (2001). Expanding learning at work: Toward an activity theoretical recon-ceptualization. *J. of Education and Work* 14(1), 133-156

9. Erickson, T. (2000). Lingua Francas for design: sacred places and pattern languages. In *Proc. of the 3rd conf. on DIS*: processes, practices, methods, and techniques (pp. 357-368). ACM.

10. Guy, E.S. (2004). *Appropriating patterns for the activity theory toolkit*. In: ATIT 2004, Inter. Workshop on Activity Theory Based Practical Methods for IT Design, 2-3 Sep 2004, Copenhagen, Denmark.

11. Martin, D., Rodden, T., Rouncefield, M., Sommerville, I., & Viller, S. (2001). Finding patterns in the fieldwork. In *Proc. of the 7th conf. on ECSCW* (pp. 39-58). Kluwer Academic Publishers.

12. McGookin, D., and Brewster, S. A. (2007) An initial investigation into non-visual computer supported collaboration. In *CHI'07: CHI'07 extended abstracts on Human factors in computing systems*, 2573-2578.

13. Metatla, O., Bryan-Kinns, N., Stockman, T., Martin, F. (2012) Supporting Cross-modal Collaboration in the Workplace. In *BCS HCI'12*, 109-118.

14. Tidwell, J. (1999) *Common Ground: A patterns language for Human-Computer Interface design*. Available from: http://www.mit.edu/~jtidwell/interaction_patterns.html (accessed: Jan. 2013).

15. Winberg, F., and Bowers, J. (2004). Assembling the senses: towards the design of cooperative interfaces for visually impaired users. In *CSCW'04*, 332-341.