<figure><img src="image_1.png" /></figure>

HAL
open science

# Learning Lessons from Controlled Studies to Investigate Users' Resilience Strategies

Jonathan Day, George Buchanan, Stephann Makri

► To cite this version:

Jonathan Day, George Buchanan, Stephann Makri. Learning Lessons from Controlled Studies to Investigate Users' Resilience Strategies. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.578-581, 10.1007/978-3-319-22723-8_65. hal-01610786

HAL Id: hal-01610786

https://inria.hal.science/hal-01610786

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Learning Lessons from Controlled Studies to Investigate
Users' Resilience Strategies

Jonathan Day, George Buchanan and Stephann Makri

Centre for HCI Design, City University London
Northampton Square, London, EC1V 0HB, UK

{Jonathan.Day.2, George.Buchanan.1, Stephann}@city.ac.uk

**Abstract.** This work describes the development and implementation of a controlled study into the way users form and utilise resilience strategies to overcome threats to performance. Despite a carefully considered design, participants demonstrated creative and unanticipated strategies to overcome deliberately 'designed-in' challenges in our task, thus circumventing the errors and responses we had predicted. We discuss the variety of unanticipated resilience strategies we observed during the course of this study, as well as methodological lessons learned as a result. Furthermore, we describe a forthcoming study which seeks to build upon the initial investigation, utilising a revised task paradigm to address and overcome its limitations.

Keywords: Resilience Strategies, Workarounds, Cognitive Resilience

# 1 Introduction

A key concern of HCI is the investigation of error:- how and why errors occur, and how they can be prevented in future interactions. An alternative, complimentary perspective however, is the investigation of Resilience Strategies, which constitute positive behavioural adaptations by users to reduce or mitigate threats to performance.

The study of resilience also presents new opportunities to improve systems and interactions, given that historically, it has largely been overlooked [1]. When errors or adverse events occur, the traditional model is to consider the frailties and causal factors that contributed to such, in an attempt to design interventions to reduce the likelihood of future occurrence. However, when resilient interventions pre-empt or neutralise a threat, thus avoiding an adverse event, analysis is seldom deemed necessary.

While the concept of resilience is primarily considered as a *property* of sociotechnical systems, an emerging reconceptualisation of the term considers 'cognitive resilience' or the resilience strategies of users interacting with systems. Such strategies include creating cues to assist memory, appropriating items, ensuring resource availability, and so fourth. This work addresses these phenomena, exploring how users develop and deploy tactics to mitigate risk, minimise error and improve performance.
## 2 Previous Work

As noted, the targeted investigation of users' resilience strategies in HCI is relatively lacking. There are some parallels between this topic and workarounds in a HCI context (e.g. [2]), however we draw distinctions based on both coverage and consequence. Resilience includes many prospective, anticipatory interventions, as opposed to the generally responsive nature of workarounds. Moreover, where (particularly in safety critical domains), workarounds are generally deemed to be unconducive to safe practice, resilience is inherently by its definition a positive contributor to outcome.

Furniss et al have however described specific work in this area, and note that existing strategy-specific work largely constitutes anecdotal discussion, with little to frame or situate such accounts. This forms the rationale for their Resilience Markers Framework [3]. Furniss et al also offer a categorisation scheme to collate and consider themes across strategies, serving to facilitate discussion and analysis [4].

While such work provides an insightful account of resilience as observed, efforts to empirically elicit examples of resilience or operationalise the concept in order to establish predictive power are seemingly not available to date. We present here an initial step towards achieving these objectives.

## 3 A Controlled Study into Cueing Strategies

As an early step toward isolating and investigating resilience strategies in a controlled setting, and owing to the apparent diversity of strategies, we initially opted to limit our scope to focus upon cueing-related strategies. Building on existing work [5,6] that suggests users not only utilise, but in some cases develop or appropriate cues to assist prospective memory, we sought to investigate the effectiveness of user-configured cues in reducing placekeeping errors during a challenging and interrupted task.

### 3.1 Study Design & Hypotheses

We designed an independent samples study, with the dependent variable consisting of error rate across three independent variables; IV1: user-configured cues, IV2: system-incorporated cues, and IV3: no cue support. IV2 enabled comparisons against a baseline where cues had been designed into the interface, and IV3 was a control where no cues were intended to be available. Our primary one-tailed hypothesis was that user-configured cues (IV1) would reduce error rates compared to the absence of cues (IV3). As a secondary, two-tailed hypotheses, we also anticipated a performance different between user configured cues (IV1) and cues hard-coded into the system (IV3).

### 3.2 Task Paradigm

The task composed for the study was modelled on a common, interruption-prone and HCI-relevant task within the medical domain: setting up the delivery of multiple medications. The task involved performing a number of relatively simple calculations (of
a [speed = distance / time] nature, but for values: rate, volume and duration) followed by data entry, transcribing data from paper to onscreen form. The different conditions were represented in this interface, with IV1 featuring unpopulated arbitrary check-boxes that users could appropriate as cues, IV2 consisting of automated visual cues of a similar size and nature, and IV3 featuring no such progress-tracking UI elements.

In an attempt to isolate engagement with and effectiveness of these cues, we attempted to ‘design out’ other implicit cues in the interface. For example, participants were trained to confirm each value immediately upon entering it, which cleared the corresponding field (so values left in fields couldn’t be used as a progress tracking cue). We also removed the standard blinking text-field cursor from the interface after a few seconds upon each field being selected, so this wasn’t available as a cue.

During the task, participants were interrupted with a paper-based distractor task (checking completed calculations). The points of interruption were predetermined, and as with the design of the onscreen form, increased the threat of place-keeping error. Data capture was via automated logging of input, remote screen monitoring and recording, collection of paper sheets, and brief informal ad-hoc questioning upon completion. Finally, prior to the study proper, a limited pilot with users unfamiliar with the research was undertaken to elicit feedback on the task and instruction.

# 4 Findings and Observations

The task performance data did not reveal any relationships between condition (form of cue) and error rate, thus leading to the rejection of the experimental hypotheses. There were however two notable insights provided from this study, one regarding the conduct and resilience of our participants, and one related to setup and task paradigm.

*Participants can be highly resilient, even when they're not expected to be.*

One key insight was that despite our best efforts to carefully control available cues, participants were very proactive in establishing alternative and additional cueing behaviours and other strategies. This enabled them to maintain performance and manage the threat presented by interruptions. Participants were, to put it simply, far more resilient than we had anticipated, and found novel and innovative ways of coping when critical threats to their cognitive working capacity were presented.

In terms of cueing, such strategies included utilising or marking paper sheets as external physical artefacts to track progress (n=13), restructuring the sequence of data entry into the onscreen form (resulting in values serving as implicit visual cues) (n=11), and using unanticipated digital artefacts as visual cues (e.g. the mouse pointer, or placing temporary and arbitrary values in fields; n=3). We also observed other types of potential resilience strategy, including the momentary deferring of experimenter interruptions (n=14), and intentional verbal rehearsal to assist memory (n=6).

*Pitching the complexity of the task paradigm is key.*

Another unanticipated insight was the significant level of individual variation of task performance between subjects, with total errors per participant ranging from 0 up to
13. This made it unfeasible, given the sample size available (n=29), to establish a baseline level of performance for each of the three groups.

While there is insufficient information to determine why this significant variation in performance occurred, we postulate this was largely a reflection of natural variation in subjects’ dexterity with figures. Both the primary and distractor tasks involved calculation, and it was noted that some participants completed such exercises with relative ease using mental arithmetic, while others relied heavily on the provided calculator and appeared to find this aspect of the task more challenging.

# 5 Lessons Learned & Revised Study

The study showed that users may be more resilient than we give them credit for. Even in a tightly constrained task, users coped well and deployed unanticipated resilience strategies, a finding which has implications for future work. In our next study we will strive not to exclude potential unforeseen resilience strategies, but will be better equipped to recognise and account for them, and potentially capture more information to enable more targeted and rigorous analysis, providing additional insights.

Another important consideration would be to ensure a stable baseline performance rate could be established in a revised task paradigm, by reducing the level of variance in terms of individual differences. We propose addressing this consideration in three ways: (i) We will avoid the use of a numerical task, and are instead moving towards a paradigm where cognitive load is introduced by workload structuring and interleaving, and subtask sequencing (ii) we will seek to establish and participants’ abilities and baseline performance in a screening activity, enabling us to control for variance, and finally (iii) we will employ more stringent and extensive piloting to ensure such a baseline in performance can be established, and will adjust task parameters as necessary to achieve this prior to execution of the study phase.

# 6 Acknowledgements

This work was supported by UK EPSRC grant [EP/G059063/1], CHI+MED.

# 7 References

1. Hollnagel, E.: A tale of two safeties. Nuclear Safety and Simulation 4 (1), 1–9 (2013)

2. Kobayashi, I.M., Fussell, S.R., Xiao, Y., Seagull, F.J.: Work co-ordination, workflow, and workarounds in a medical context. Proc. CHI 2005. Portland, Oregon, USA. (2005)

3. Furniss, D., Back, J., Blandford, A., Hildebrandt, M., Broberg, H.: A resilience markers framework for small teams. Reliability Engineering & System Safety, 96(1), 2–10 (2011)

4. Furniss, D., Back, J., Blandford, A.: Cognitive Resilience: Can we use Twitter to Make Strategies More Tangible? Proc. ECCE 2012. (2012)

5. Back, J., Furniss, D., Hildebrandt, M., Blandford, A.: Resilience Markers for Safer Systems and Organisations. Proc. SAFECOMP 2008, 99–112 (2008)

6. Dix, A.: Designing for Appropriation. Proc. BCS HCI 2007. 2, 28-30 (2007)