# Pilot Implementation: Testing Human-Work Interaction Designs

Morten Hertzum, Torkil Clemmensen, Barbara Rita Barricelli, Pedro F. Campos, Frederica Gonçalves, José Abdelnour Nocera, Ganesh Bhutkar,

Arminda Guerra Lopes

► To cite this version:

Morten Hertzum, Torkil Clemmensen, Barbara Rita Barricelli, Pedro F. Campos, Frederica Gonçalves, et al. Pilot Implementation: Testing Human-Work Interaction Designs. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.570-574, 10.1007/978-3-030-85607-6_79. hal-04291193

HAL Id: hal-04291193

https://inria.hal.science/hal-04291193v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Pilot Implementation: Testing Human-Work Interaction Designs

Morten Hertzum<fnref n="1" />[0000-0003-0019-8531], Torkil Clemmensen<fnref n="2" />[0000-0002-0934-2336], Barbara Rita Barricelli<fnref n="3" />[0000-0001-9575-5542], Pedro F. Campos<fnref n="4" />, Frederica Gonçalves<fnref n="4" />, José Abdelnour Nocera<fnref n="4" />,<fnref n="5" />, Ganesh Bhutkar<fnref n="6" />[0000-0002-3044-6471], and Arminda Guerra Lopes<fnref n="4" />,<fnref n="7" />

**Abstract.** Pilot implementations are field tests of properly engineered, yet unfinished, systems. In contrast to lab tests, the users in a pilot implementation use the system for performing real work. In contrast to full-scale implementations, the objective of a pilot implementation is to learn. The workshop on pilot implementation aims to (a) help mature this technique for evaluating human-work interaction designs during the process of their development and implementation, (b) collect case studies that analyze experiences with conducting and learning from pilot implementations, and (c) formulate a research agenda for future work on pilot implementations – addressing their strengths, limitations, conduct, impact, and so forth. The target audience for the workshop is researchers and practitioners working on topics related to work analysis, interaction design, system-organization fit, organizational implementation, benefits realization, and in-the-wild evaluation.

**Keywords:** Pilot implementation, Field test, Human-work interaction design.

# 1 Introduction

The integration of work analysis and interaction design methods is pivotal to the successful development and implementation of systems for pervasive and smart workplaces [13]. With this workshop, the IFIP TC13.6 working group on Human Work Interaction Design (HWID) aims to create a forum for investigating and discussing how

<footnote n="1">¹ University of Copenhagen, Karen Blixens Plads 8, DK-2300 Copenhagen, Denmark
hertzum@hum.ku.dk</footnote>

<footnote n="2">² Copenhagen Business School, Howitzvej 60, DK-2000, Frederiksberg, Denmark
tc.digi@cbs.dk</footnote>

<footnote n="3">³ Università degli Studi di Brescia, Via Branze 38, 25123 Brescia, Italy
barbara.barricelli@unibs.it</footnote>

<footnote n="4">⁴ ITI/LARSyS and University of Madeira, Funchal, Portugal
pedro.campos@iti.larsys.pt, frederica.goncalves@iti.larsys.pt</footnote>

<footnote n="5">⁵ University of West London, London, UK
Jose.Abdelnour-Nocera@uwl.ac.uk</footnote>

<footnote n="6">⁶ Vishwakarma Institute of Technology, Pune, India
ganesh.bhutkar@vit.edu</footnote>

<footnote n="7">⁷ Polytechnic Institute of Castelo Branco, Portugal
aguerralopes@gmail.com</footnote>
pilot implementations can – and cannot – contribute to this integration. The workshop builds on prior research documenting the value of in situ methods in working to ensure and improve system usefulness [e.g., 3, 6, 14].

A pilot implementation is “a field test of a properly engineered, yet unfinished system in its intended environment, using real data, and aiming – through real-use experience – to explore the value of the system, improve or assess its design, and reduce implementation risk” [7]. This definition positions pilot implementation in between predictive usability tests and full-scale implementation. In contrast to usability tests, which are conducted in the lab to evaluate mockups or prototypes, pilot implementations are conducted in the field and necessarily involve users performing their real work using the pilot system. In contrast to full-scale implementation, which is intended to result in continued use, pilot implementations are temporary and aim to feed information about the finalization and implementation of the system back into the development process.

Pilot implementations recognize that systems are sociotechnical and that their social, organizational, and contextual qualities can only be evaluated thoroughly in the field. This recognition of a need for extending the iterative design process beyond the lab is not exclusive to pilot implementations. It is also at the fore in, for example, design in use [4], living labs [1], technology probes [9], digital twins [12], and breaching experiments [5]. However, some of these other methods are mainly used by researchers who study technology use. In contrast, pilot implementations are widely used by practitioners within the resource limitations of development and implementation processes.

## 2 Workshop Objectives

The HWID workshop on pilot implementation has three main objectives:

* To help mature pilot implementation as a technique for evaluating human-work interaction designs during the process of their development and implementation.

* To collect case studies that analyze experiences – good and bad – with conducting and learning from pilot implementations.

* To formulate a research agenda for future work on pilot implementations – addressing their strengths, limitations, conduct, impact, and so forth.

## 3 Pilot Implementation

Prior work emphasizes that a pilot implementation is not just the period during which a system is in pilot use. Hertzum et al. [7] propose that pilot implementation involves five interrelated activities:

* *Planning and design*, during which the pilot implementation is defined. This activity involves deciding where and when the pilot implementation will take place, how lessons learned during the pilot implementation will be collected, and so forth.
* *Technical configuration*, during which the pilot system is configured for the pilot site. As part of this activity, data must be migrated to the system and interfaces to other systems at the pilot site must be developed or simulations set up.

* *Organizational adaptation*, during which the pilot site revises work procedures to benefit from the pilot system. This activity also involves user training and, possibly, temporary safeguards against pilot-system breakdowns.

* *Use*, during which the system is used for real work at the pilot site. This activity involves striking a balance between integrating the system in normal procedures and maintaining a focus on the system as an object under evaluation.

* *Learning*, which is the overarching objective of pilot implementation. This activity runs in parallel with the four other activities and collects data and insights about the system, its use, and the associated organizational changes.

The three first activities are preparations, which may consume more time than the period of pilot use. During the period of pilot use, the consequences of the system become salient to the users because it begins to influence their daily work and requires them to change their practices. This salience motivates the users to voice their concerns [15]. They will often be more motivated to participate at this stage than during requirements specification and usability tests, which may be experienced as a distraction that takes time away from getting their daily work done. Finally, learning occurs during the preparations as well as during the period of pilot use. The collection and analysis of the learnings require work. This work may involve measuring whether specified benefits are realized, interviewing users about their user experience, and having managers discuss additional visions for changes associated with the system.

Pilot implementations create room for innovative experimentation [2], help align the stakeholders on whom system adoption is dependent [11], provide valuable feedback about system finalization [7], identify important discrepancies in how different user groups perceive a system [16], and create a decision point for discontinuing systems that face too many obstacles [8]. These strengths are important arguments for conducting pilot implementations. However, multiple challenges have also been identified, including the following [7, 8, 10, 11, 16]:

* Pilot implementations tend to be either too early or too late. When they are too early, the system and organization are not yet ready. When they are too late, the benefits to be realized from full-scale implementation are unduly delayed.

* The boundaries of pilot implementations are continuously negotiated because any choice comes with difficult-to-handle interactions across the boundary between the pilot site and its surroundings, which have not yet implemented the system.

* The focus on learning is difficult to maintain in the midst of getting the daily work done. Some users may see the pilot implementation as the first stage of full-scale implementation and fail to notice the learning objective altogether.

* Pilot implementations involve special precautions to safeguard against critical errors and to encourage system use. There is a tension between the presence of these precautions and their absence when the system is released for full-scale use.
* The learning may be situated and difficult to transfer from the pilot site to other sites. Specifically, it is difficult to distinguish consequences that are local to the pilot implementation from consequences that are inherent in the system and its use.

* Pilot implementations are not final statements about the fit between system and organization. They add realism (compared to lab tests), but the realism increases salience rather than terminates discussion about what using the system will be like.

* Pilot implementations are generative; they are not merely tests. A pilot implementation fosters alignment, reveals tension, shifts power, and otherwise affects users and organizations. All these effects must be contended with post pilot.

Pilot implementations are recognized among practitioners for their contributions to system finalization, organizational implementation, and project de-escalation. However, the extant research on pilot implementations leaves lots of open questions about common experiences with conducting pilot implementations, about how best to organize them, about their pitfalls, and about which questions the research on pilot implementation needs to address.

## 4 Expected Outcomes

The workshop will produce a research agenda for studying pilot implementations and how best to conduct them. The aim of this research agenda is to stimulate further research interest and provide direction for research on pilot implementation. At the workshop, the organizers will invite the workshop participants to co-author a paper that presents the research agenda and discusses it on the basis of the cases and insights contributed by the participants.

## 5 Target Audience

The target audience for the HWID workshop on pilot implementation includes researchers and practitioners working on topics related to work analysis, interaction design, system-organization fit, organizational implementation, benefits realization, and in-the-wild evaluation. Because pilot implementations are common in practical development and implementation projects, we pay special attention to attracting practitioners from various work environments to discuss real-life case studies. Participation in the workshop requires the submission and acceptance of a position paper, which is limited to a maximum of four pages. Early-stage researchers and PhD students are encouraged to submit papers describing work in progress.

## 6 Organizing Committee

The workshop is organized by IFIP TC13 WG6 - Human Work Interaction Design (https://barbara-barricelli.unibs.it/HWID). The organizers are as follows:
*Barbara Rita Barricelli* is assistant professor at the Department of Information Engineering of Università degli Studi di Brescia, Italy. Her research interests lie in the field of Human-Computer Interaction, and specifically: End-User Development, Interaction Design, and Human Work Interaction Design. She is Chair of the IFIP TC13.6 working group on Human Work Interaction Design.

*Ganesh Bhutkar* is assistant head (research) at Department of Computer Engineering at Vishwakarma Institute of Technology (VIT), Pune, India. He has a PhD from Indian Institute of Technology, Bombay. He has established a Centre of Excellence in HCI at VIT. His research mainly focuses on Assistive Technologies and Medical Usability. He is Vice-Chair of IFIP WG 13.6 on Human Work Interaction Design.

*Pedro F. Campos* is associate professor at the University of Madeira, Portugal, and vice-president for research at ITI (Interactive Technologies Institute), LARSys. His research interests span human-computer interaction, persuasive computing, virtual reality, and human work interaction design. He is a co-founder of IFIP TC13 WG6.

*Torkil Clemmensen* is professor at the Department of Digitalization, Copenhagen Business School, Denmark. His research interest is in psychology as a science of design. His research focuses on cultural and psychological perspectives on usability, user experience, and the digitalization of work. He contributes to Human-Computer Interaction, Design, and Information Systems. He is a co-founder of IFIP TC13 WG6.

*Frederica Gonçalves* is assistant professor at University of Madeira and researcher at ITI/LARSYS, Portugal. Her research interests are in HCI, Creativity, Cognitive Augmentation, Persuasive Computing, Creativity Support Tools, and Human Work Interaction Design. She is Secretary Officer of IFIP TC13 WG6.

*Morten Hertzum* is professor of Information Science at University of Copenhagen, Denmark. His overall research interest concerns how information technology supports, and otherwise affects, human activity. He pursues this interest within human-computer interaction, computer-supported collaborative work, healthcare informatics, and implementation studies.

*Arminda Guerra Lopes* is professor at Polytechnic Institute of Castelo Branco, Portugal, and research fellow at ITI/LARSys, Portugal. Her research interests are in Human-Computer Interaction and research methodologies. The focus is on Social Informatics, Interaction Design, Human Work Interaction Design, Creativity and Innovation, Collaborative Work, and Quality of Life Technologies.

*José Abdelnour Nocera* is professor in Sociotechnical Design and Head of the Sociotechnical Centre for Innovation and User Experience at the University of West London. He is the current Chair for the British Computer Society Sociotechnical Specialist Group. His interests lie in the sociotechnical and cultural aspects of systems design, development, and use.

## References

1. Alavi, H.S., Lalanne, D., Rogers, Y.: The five strands of living lab: A literature study of the evolution of living lab concepts in HCI. *ACM Transactions on Computer-Human Interaction* 27(3), article 10 (2020).
2. Bakker, R.M., Cambré, B., Korlaar, L., Raab, J.: Managing the project learning paradox: A set-theoretic approach toward project knowledge transfer. *International Journal of Project Management* 29(5), 494-503 (2011).

3. Bannon, L.J.: From human factors to human actors: The role of psychology and human-computer interaction studies in system design. In: Greenbaum, J., Kyng, M. (eds.) *Design at Work: Cooperative Design of Computer Systems*, pp. 25-44. Erlbaum, Hillsdale, NJ (1991).

4. Bjögvinsson, E., Ehn, P., Hillgren, P.-A.: Design things and design thinking: Contemporary participatory design challenges. *Design Issues* 28(3), 101-116 (2012).

5. Crabtree, A.: Taking technomethodology seriously: Hybrid change in the ethnomethodology-design relationship. *European Journal of Information Systems* 13(3), 195-209 (2004).

6. Fields, B., Amaldi, P., Wong, W., Gill, S.: In use, in situ: Extending field research methods. *International Journal of Human-Computer Interaction* 22(1&2), 1-6 (2007).

7. Hertzum, M., Bansler, J.P., Havn, E., Simonsen, J.: Pilot implementation: Learning from field tests in IS development. *Communications of the Association for Information Systems* 30(1), 313-328 (2012).

8. Hertzum, M., Manikas, M.I., Torkilsheyggi, A.: Grappling with the future: The messiness of pilot implementation in information systems design. *Health Informatics Journal* 25(2), 372-388 (2019).

9. Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B.B., Druin, A., Plaisant, C., Beaudouin-Lafon, M., Conversy, S., Evans, H., Hansen, H., Roussel, N., Eiderbäck, B., Lindquist, S., Sundblad, Y.: Technology probes: Inspiring design for and with families. In: *Proceedings of the CHI2003 Conference on Human Factors in Computing Systems*, pp. 17-24. ACM Press, New York (2003).

10. Korn, M., Bødker, S.: Looking ahead - How field trials can work in iterative and exploratory design of ubicomp systems. In: *Proceedings of the UbiComp2012 Conference on Ubiquitous Computing*, pp. 21-30. ACM Press, New York (2012).

11. Mønsted, T., Hertzum, M., Søndergaard, J.: A socio-temporal perspective on pilot implementation: Bootstrapping preventive care. *Computer Supported Cooperative Work* 29(4), 419-449 (2020).

12. Nikolakis, N., Alexopoulos, K., Xanthakis, E., Chryssolouris, G.: The digital twin implementation for linking the virtual representation of human-based production tasks to their physical counterpart in the factory-floor. *International Journal of Computer Integrated Manufacturing* 32(1), 1-12 (2019).

13. Nocera, J.A., Barricelli, B.R., Lopes, A., Campos, P., Clemmensen, T. (Eds.). *Human work interaction design: Work analysis and interaction design methods for pervasive and smart workplaces* (Vol. IFIP AICT 468). Springer, Cham (2015).

14. Nocera, J.A., Dunckley, L., Sharp, H.: An approach to the evaluation of usefulness as a social construct using technological frames. *International Journal of Human-Computer Interaction* 22(1&2), 153-172 (2007).

15. Wagner, E.L., Piccoli, G.: Moving beyond user participation to achieve successful IS design. *Communications of the ACM* 50(12), 51-55 (2007).

16. Winthereik, B.R.: The project multiple: Enactments of systems development. *Scandinavian Journal of Information Systems* 22(2), 49-64 (2010).