# How Do Users Perceive a Design-in-Use Approach to Implementation? A Healthcare Case

Morten Hertzum, Arnvør Torkilsheyggi

► To cite this version:

Morten Hertzum, Arnvør Torkilsheyggi. How Do Users Perceive a Design-in-Use Approach to Implementation? A Healthcare Case. 17th IFIP Conference on Human-Computer Interaction (INTER-ACT), Sep 2019, Paphos, Cyprus. pp.410-430, 10.1007/978-3-030-29387-1_23. hal-02553878

HAL Id: hal-02553878

https://inria.hal.science/hal-02553878

Submitted on 24 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# How Do Users Perceive a Design-in-Use Approach to Implementation? A Healthcare Case

Morten Hertzum<fnref n="1" />[0000-0003-0019-8531] and Arnvør Torkilsheyggi<fnref n="2" />

hertzum@hum.ku.dk

**Abstract.** The implementation of information systems in organizational settings is a protracted process that includes the mutual adaptation of system and organization to each other after the system has gone live. We investigate a design-in-use approach to this implementation process. Rather than a centrally run implementation process with preset goals, the management in the studied hospital tasked the individual departments with exploring and embracing the possibilities afforded by a network of interconnected electronic whiteboards. The responsibility for driving this process was assigned to local super users in the departments. On the basis of interviews with 17 clinicians we find that (a) they perceive the design-in-use approach in conflicting ways, (b) the super users are more positive about the approach than the end-users, (c) standardization across departments conflicts with design in use within departments, (d) intradepartmental change is perceived more positively, (e) the design-in-use process is inextricably sociotechnical, and (f) the clinicians' perception of design in use is more about implementing change than about preparing it or about training and support. The conflicting perceptions of the design-in-use approach, for example, include whether it gained momentum, met local needs, and made for an engaging process. We discuss the implications of our findings for a design-in-use approach to implementation.

**Keywords:** Design in use, Organizational implementation, Healthcare.

## 1 Introduction

The implementation of a new information system in an organization temporarily disturbs work and normally leads to a productivity dip in the period following go-live [26, 35]. While a quick return to baseline productivity suggests a well-executed implementation process, it also incurs the risk that work practices with the new system congeal too quickly. If so, the organization will not reap the full benefit of the new system [44]. To reap the full benefit it is necessary for the organization to experiment with the possibilities afforded by the system and to seize the opportunities that emerge from this experimentation [38]. Such a process requires that the users are prepared to prolong the period in which work practices are fluid in order to spend time exploring what new ways of working the system affords. That is, it requires that

<footnote n="1">¹ University of Copenhagen, Copenhagen, Denmark</footnote>

<footnote n="2">² Roskilde University, Roskilde, Denmark</footnote>
the users are prepared to engage in a design-in-use process. We investigate this preparedness.

The empirical setting for our investigation is the organization-wide implementation of a network of interconnected electronic whiteboards for intra- and interdepartmental coordination at a hospital. Management in the studied hospital took a design-in-use approach to implementation. Rather than a centrally run implementation process that stipulated up front how to use the whiteboard, management tasked the individual departments with exploring and embracing the possibilities afforded by the whiteboard. This design-in-use approach to implementation signaled a commitment to reaping benefit from the whiteboard, even if it meant prolonging the implementation process. However, it also meant that successful implementation became dependent on the users' preparedness to design their use of the whiteboard rather than simply use it according to preset procedures. Specifically, the group of super users was assigned a key role as drivers of the local implementation process. We interviewed 17 clinicians about their thoughts on the implementation process. On that basis we seek to answer two research questions:

* *How do the users perceive the design-in-use approach to implementation?* To answer this question we make a content analysis of the interviewees' statements and catalog the opportunities and barriers they mention.

* *Do the users' perceptions vary across user groups?* The answer to this supplementary question consists of comparing the distribution of the content categories for super users and end-users and for physicians and nurses.

In the following we review related work, account for our method of data collection and analysis, present the results of the study, and discuss their implications.

# 2 Related Work

Research on design in use aims to capture “practices of interpretation, appropriation, assembly, tailoring and further development of computer support in what is normally regarded as deployment or use” [9, p. 125]. This definition suggests that design in use is a fairly informal and largely user-driven set of practices. However, the definition also implies a permeable boundary between design in use and the more planned and management-initiated activities of system implementation.

## 2.1 Design in Use

Orlikowski [28] shows that the implementation of information systems is only a partially planned process. While part of the process can, and should, be planned in advance and executed as planned, other parts emerge in unplanned ways during use and become visible only in retrospect. These emergent parts of implementation present and deny opportunities that may be pursued in additional, opportunity-based efforts to obtain benefit from a system or may call for revising plans to avoid adverse side effects [14, 29]. As a result, the implementation process continues over an
extended period of time; it is not restricted to the first few weeks after a system has gone live. Design-in-use research embraces this continuation by contending that go-live does not mark a shift from design to use [1, 5, 8, 10, 12, 16, 18, 19, 24, 41, 42]. Rather, design continues into use in the sense that users design, as opposed to merely adopt, their ways of working with a new system. The continued design activities are informed by the users' experiences from using the system for real work and may involve configuring the system as well as adjusting work practices.

The boundary between design in use and system implementation depends largely on the extent to which design in use proceeds as an informal process among peers or has become a more formalized process with organizationally defined roles and responsibilities. Organizational structures for deciding and coordinating which changes to pursue become increasingly needed with systems that are still more configurable and thereby make it possible to pursue a still wider range of changes [8]. With increasing managerial support design in use increasingly becomes an approach to systems implementation. We distinguish three broad classes of design-in-use processes with respect to managerial support.

First, multiple studies have investigated design in use as it unfolded in situations with no formal organizational support. For example, Mackay [20] studied how customizations were shared among the users of an application package. While most users made some customizations to adapt the package to their needs and desires, a small group of users spent considerable time making customizations and became adept at it. The members of this group became the go-to persons when their colleagues had questions about customization; furthermore, the group members started to share useful customizations. Although this sharing was widespread “few of the managers were aware that customization files were being exchanged and none were aware of the extent of sharing” [20, p. 219]. Thus, these design-in-use activities took place without organizational support. Relatedly, Park et al. [30] analyzed how doctors started making notes on pieces of paper when an electronic medication record required them to remember information they gathered at the bedside until they later visited the charting room to enter it in the electronic record. This apparently mundane example of design in use remained organizationally unrecognized until it was realized that the notes necessitated further design in use because they contained sensitive patient information and, therefore, had to be discarded in a safe manner.

Second, design in use has been studied in settings with organizational support for the users' tailoring activities. In several of these cases design in use started as an informal activity performed by technically minded individuals for personal purposes and only gradually received organizational recognition and support [e.g., 31, 39, 42]. While many of these design-in-use activities have produced valued extensions to base functionality, others have salvaged the base functionality by contributing “the work to make IT work” [7, p. 296]. Multiple studies find that the available organizational support for design-in-use activities tends to be inadequate. For example, Dittrich et al. [9] conclude that a better infrastructure for supporting design in use is necessary to achieve the vision declared for the studied municipal website. They base this need on the uneven availability of developer support for design-in-use activities combined with the high organizational value of the templates developed by a user who had
access to support. At the same time, Spencer [39] documents user frustration with the organizational practices introduced to support design in use. The source of the frustration was that the practices were bureaucratic and prioritized long-term evolution over here-and-now changes. The kind of support emphasized by these authors is technical. In contrast, Hartswood et al. [11] point to a need for support in devising, or repairing, the work processes that surround new systems. In their study the evolving use of a speech-recognition system shifted additional workload to the secretaries, who had to handle speech-recognition errors left uncorrected by the clinicians. After the secretaries raised this problem, the ward instituted a work procedure where the secretaries returned letters with uncorrected errors to the clinicians. This procedure regulated the collaborative use of the opportunities afforded by the system and rebalanced the workload.

Third, design in use may be the planned approach to implementation. In their review of the factors critical to implementation success, Nah et al. [25] emphasize issues such as exploiting the best practices offered by the system, fitting work practices to the system to minimize the need for customizations, and championing the system to manage resistance. These issues are largely directed at realizing the benefits that were planned ahead of go-live. An exclusive focus on planned change has been criticized by Orlikowski and Hofman [29], who instead emphasize the importance of identifying and embracing emergent change. As a result, design in use is central to their approach to improvisational change management. Design in use serves the double purpose of responding to local circumstances in a manner that pursues planned ends and seizes the opportunities provided by emergent change [14]. To fulfill this purpose Markus [21] stresses the importance of troubleshooting and shakedown in the period immediately after go-live; Karasti et al. [16] show the need to sustain design-in-use processes for years or even decades; Yetim et al. [45] propose to extend systems with an embedded tool for collecting design-in-use ideas; and Hertzum and Simonsen [15] catalog the competences needed locally to be able to configure systems and work practices for each other.

## 2.2 Perceptions of Design in Use

Users expect that their systems can be tailored to their needs, but to exploit the possibilities afforded by a system local practices must also be adapted to the system [3, 4]. Both components of this mutual adaptation of system and practices to each other are candidates for design in use but they may be perceived differently. Users tend to appreciate design in use – by embracing it themselves or valuing their colleagues’ design-in-use efforts – when systems are adapted to local needs and it is voluntary whether to adopt the adaptations [20, 42]. When design in use is integral to the planned approach to implementation and, thereby, at least partially mandatory then the perception of design in use becomes dependent on the available support [1, 34]. For example, Åsand and Mørch [46] find that end-users mostly prefer to receive support from super users with local and domain knowledge, rather than from technical IT staff. Other studies identify a need for support from allies with the organizational power to ensure that local design-in-use solutions are spread and adopted in the
organization [27, 41]. However, the main issue with respect to support is, probably, that the available support is often perceived as insufficient and, thereby, as constraining the possibilities for design in use [e.g., 8, 11, 24].

Several studies report tensions that affect the users' perception of design in use negatively. For example, Karasti et al. [16] report that the users struggled to maintain a focus on evolving the system in accordance with their long-term needs because the developers who supported design in use had a shorter temporal perspective. Torkilsheyggi and Hertzum [41] report a tension between management's expectations to the design-in-use process and the goals pursued in the local design-in-use efforts. This tension was aggravated by simultaneous, but uncoordinated, evolution in management expectations and locally pursued goals. In contrast, other studies report from design-in-use processes that were positively perceived and led to recognized improvements in local practices. For example, the Norwegian rehabilitation hospital Sunnaas has implemented and evolved videoconferencing through a deliberate design-in-use process that has spanned two decades [1]. This process has, among other things, introduced follow-up consultations in which the patients participate from their home via videoconference. In Germany, the POLITeam project improved the process of vote preparation [32]. The improvement opportunity was, however, not planned ahead but realized "rather accidentally" [32, p. 207] after the system had been in use for several months and, then, led to a redesign of the workflow. At a Dutch hospital a new computerized physician order entry system succeeded because the nurses, unexpectedly, adopted the system to document nursing care [2]. When it became apparent that the physicians – the originally intended users of the system – would not adopt it, the system was revised to accommodate its new users and the workflow was revised by allowing the physicians to authorize orders by signing printouts from the system.

Mehandjiev et al. [22] investigated how end-user development (EUD) was perceived by EUD researchers, IT managers, and end-users. EUD comprises the tools and activities that seek to "enable non-software specialists to develop nontrivial software artefacts to support their work" [22, p. 371]. It, thus, constitutes an approach to the technical component of design in use. In terms of advantages the participants perceived that EUD could speed up software development, make users more efficient in their main job tasks, make users' work more interesting, and increase the agility of organizations in responding to external market pressures. The perceived risks included that the users might not be motivated, that the learning curve might be too steep, and that the developed software might be low quality. With respect to quality, it has been found that errors are prevalent in EUD software [33] and that end-users are often overconfident in the correctness of their software [17]. Mehandjiev et al. [22] propose that EUD software should be audited by professional developers to bolster quality, but they also recognize that auditing is somewhat at odds with EUD, which stimulates an informal process. Relatedly, Mørch and Andersen [24] present a process of mutual development, in which end-users and professional developers collaborate on making changes after go-live. Through this mutual process the development organization gets customer input for relevant changes and the end-users get more advanced changes than they would be able to develop on their own.
# 3 Method

This study was based on interviews. Prior to the interviews the study was approved by hospital management. The interviewees individually consented to take part.

## 3.1 Setting

The study setting was a hospital in Region Zealand, one of the five healthcare regions in Denmark. The hospital had 250 beds and about 35,000 annual admissions. We have been following the implementation and use of the network of electronic whiteboards at the hospital since the first whiteboards were introduced in the emergency department in 2009 [36]. Because the whiteboard was a success in the emergency department [13] it was introduced on all departments in the hospital in December 2012. The overall purpose of the whiteboard was to support coordination in and among the departments by sharing information about the status and flow of the patients and by providing at-a-glance access to selected information from the electronic patient record. However, management adopted a design-in-use approach in the hospital-wide implementation of the whiteboard [41]. That is, the individual department was tasked with exploring and embracing the specific possibilities afforded by the whiteboard in the department. To drive this process of configuring the whiteboard and the clinical work for each other, super users were appointed in each department. The whiteboard was accessible on all computers and permanently shown on large wall-mounted displays, see Fig. 1. It gave one row of information for each patient. This information might, for example, include time of arrival, name, room, responsible physician, status of laboratory tests, and a transfer checklist.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Clinicians meeting by the whiteboard.</figcaption></figure>

Configuring the whiteboard for a department involved defining views that showed different subsets of the patients, choosing the fields of information to appear in each view, and facilitating the incorporation of the views in the clinicians’ work practices.
A prominent example of an outcome of the design-in-use process was the reorganization of the communication between the general wards and the team of physiotherapists. This interdepartmental change started by the physiotherapists' decision to require that physiotherapy was ordered on the whiteboard to improve their possibilities for planning their work. Physiotherapy had previously been ordered on the phone or face to face. The general-ward nurses were, instead, to write the orders in the notes field on the whiteboard in their department; the team of physiotherapists would monitor the corresponding field on their whiteboard. To enforce the change the physiotherapists subsequently stopped attending the morning meetings at the general wards because the general-ward nurses continued to make face-to-face orders during these meetings. Gradually, the physiotherapists also began to provide their conclusions from completed physiotherapy as notes on the whiteboard because these notes were noticed immediately on the general wards, whereas notes in the electronic patient record were often not.

## 3.2 Interviewees

Because we were interested in the clinicians' perception of the design-in-use process we chose interviews as our method of data collection. We interviewed 17 super users, end-users, and local managers, see Table 1. All interviewees were clinicians directly involved in the day-to-day performance of the work supported by the whiteboard. Thus, all of them had experienced the implementation process first hand. The interviewees represented eight of the hospital's ten clinical departments and mostly consisted of physicians and nurses, but other professional groups were also included.

<table><caption>**Table 1.** The interviewees.</caption><thead><tr><td>Job title</td><td>Role</td><td>Professional group</td><td>Department</td></tr></thead><tbody><tr><td>Healthcare assistant</td><td>Super user</td><td>Healthcare assistant</td><td>Geriatric</td></tr><tr><td>Deputy manager</td><td>Super user</td><td>Nurse</td><td>Gynecological and Obstetrical</td></tr><tr><td>Midwife</td><td>Super user</td><td>Midwife</td><td>Gynecological and Obstetrical</td></tr><tr><td>Physician</td><td>Super user</td><td>Physician</td><td>Medical</td></tr><tr><td>Nurse</td><td>Super user</td><td>Nurse</td><td>Medical</td></tr><tr><td>Head of department</td><td>Super user</td><td>Physician</td><td>Orthopedic Surgical</td></tr><tr><td>Nurse</td><td>Super user</td><td>Nurse</td><td>Pediatric</td></tr><tr><td>Secretary</td><td>Super user</td><td>Secretary</td><td>Pediatric</td></tr><tr><td>Physiotherapist</td><td>Super user</td><td>Therapist</td><td>Rheumatology</td></tr><tr><td>Occupational therapist</td><td>Super user</td><td>Therapist</td><td>Rheumatology</td></tr><tr><td>Chief physician</td><td>End-user</td><td>Physician</td><td>Emergency</td></tr><tr><td>Nurse</td><td>End-user</td><td>Nurse</td><td>Emergency</td></tr><tr><td>Physician</td><td>End-user</td><td>Physician</td><td>Geriatric</td></tr><tr><td>Deputy manager</td><td>End-user</td><td>Nurse</td><td>Orthopedic Surgical</td></tr><tr><td>Chief physician</td><td>End-user</td><td>Physician</td><td>Surgical</td></tr><tr><td>Head of department</td><td>Management</td><td>Physician</td><td>Gynecological and Obstetrical</td></tr><tr><td>Head nurse</td><td>Management</td><td>Nurse</td><td>Surgical</td></tr></tbody></table>
## 3.3 Procedure

The interviews were conducted in the fall of 2014, one and a half years after the whiteboard was introduced in all departments at the hospital. Thus, the interviewees' perceptions of the design-in-use approach to implementation were based on substantial experience. In addition, the design-in-use process was still ongoing and thereby present to mind for the interviewees. The interviewees were identified in collaboration with the system administrator for the whiteboard. He received a description of our study with a request to interview clinicians distributed across departments, staff groups, and roles in relation to the whiteboard.

After identifying the interviewees we initially contacted them by email. To give the interviewees a sense of the interview topics, the email included the guiding questions for the interview: (a) How do you, in your department, use the whiteboard, and for what? (b) How have you worked with configuring the whiteboard and incorporating it in your workflows? (c) What do you see as the biggest challenges in getting everybody in the department to use the whiteboard? (d) To what extent do you experience that the whiteboard, in its present configuration, supports intra- and interdepartmental coordination? (e) What would it take for the whiteboard to support intra- and interdepartmental coordination better? These questions covered the implementation process and the resulting use of the whiteboard from multiple angles, thereby adding nuance to the interviewees' thoughts on the design-in-use approach to implementation. The interviews lasted an average of 52 minutes.

## 3.4 Data Analysis

The data analysis involved four steps. First, we transcribed the 17 interviews verbatim from the audio recordings. Second, we read through the transcripts and identified all segments in which the interviewees made expressions about how they perceived the design-in-use approach to implementation. This step resulted in the identification of 433 segments. To maintain the context of the segments we marked them up in the transcripts rather than extracted them from the transcripts. Third, we coded the segments with respect to the six classifications in Table 2. For the classifications of valence, object, stage, and scope each segment was coded with one of the classification categories or with 'other' if none of the categories matched the content of the segment. For the classifications of opportunities and barriers we chose against preset categories and, instead, descriptively annotated the segments that expressed opportunities or barriers. This process produced 143 annotations about opportunities and 224 annotations about barriers. Fourth, we established categories of opportunities and barriers in a bottom-up manner by grouping annotations with similar content. The three last steps of the data analysis were completed by the first author alone. Thus, we cannot provide measures of inter-rater agreement.

We coded the valence of the segments to get a direct indication of the extent to which the interviewees were positive or negative toward the design-in-use approach to implementation. The balance between positive and negative valence is important to any assessment of how users perceive design in use. The rationale for the object
classification was the sociotechnical nature of design-in-use processes. This classification distinguished between information technology and work processes, while also allowing for their combined presence. The stage classification concerned how far the design-in-use process had progressed from preparing change, through implementing it, to training and support in a change that was already in place. At the studied hospital it was new that the super users became responsible for preparing and implementing change; they had long been tasked with training and supporting their colleagues. The scope classification distinguished between intra- and interdepartmental change. We included this classification because users might have the knowledge and network necessary to make them comfortable with devising change internal to their department but lack the knowledge, network or inclination to engage in the increased complexity of interdepartmental change. Finally, we included the classifications of opportunities and barriers to catalog the features that the users perceived as positive and negative, respectively, about the design-in-use approach.

<table><caption>**Table 2.** Classification categories.</caption><thead><tr><th>Classification</th><th>Category</th><th>Description</th></tr></thead><tbody><tr><td rowspan="3">Valence</td><td>Positive</td><td>In favor of a design-in-use approach to implementation</td></tr><tr><td>Negative</td><td>Against a design-in-use approach to implementation</td></tr><tr><td>Neutral</td><td>Neither for nor against a design-in-use approach</td></tr><tr><td rowspan="3">Object</td><td>Information technology</td><td>Technological aspects, such as system configuration</td></tr><tr><td>Work process</td><td>Socio-organizational aspects, such as new procedures</td></tr><tr><td>Both</td><td>Information system and work process combined</td></tr><tr><td rowspan="3">Stage</td><td>Preparing change</td><td>Analyzing situation, devising solution, planning change</td></tr><tr><td>Implementing change</td><td>Configuring system, revising processes, motivating use</td></tr><tr><td>Training and support</td><td>Scheduled learning sessions and in-situ consultation</td></tr><tr><td rowspan="3">Scope</td><td>Intradepartmental</td><td>Changes within a single department</td></tr><tr><td>Interdepartmental</td><td>Changes in the collaboration among departments</td></tr><tr><td>Both</td><td>Changes within and among departments</td></tr><tr><td>Opportunities</td><td></td><td>Positive features of a design-in-use approach</td></tr><tr><td>Barriers</td><td></td><td>Negative features of a design-in-use approach</td></tr></tbody></table>

# 4 Results

In the following we first analyze the valence, object, stage, and scope classifications, then the opportunities and barriers, and finally variations across user groups. Sections 4.1 and 4.2 address the first research question, Section 4.3 the second.

## 4.1 Perception of the Design-in-Use Approach to Implementation

Table 3 shows the distribution of the clinicians' 433 comments across the categories of the valence, object, stage, and scope classifications. With respect to *valence* the clinicians made a substantial number of positive comments about the design-in-use
approach to implementation as well as a substantial number of negative comments. The positive comments included: "I think it is wildly visionary to roll out [the whiteboard] across the entire hospital", "It is super fast to configure the whiteboard", and "It has changed our work processes ... In all sorts of ways." In contrast, the negative comments included: "We do not need to design an IT system, we need to get an IT system", "It is cumbersome and slow to get anything done and approved", and "There were all those possibilities for tailoring it yourself and every department was allowed to do that. It ended in a mess." Overall, there were slightly more negative (48%) than positive (40%) comments, and few neutral (12%). The low number of neutral comments might, in part, reflect that several of our interview questions asked the clinicians about their views on the design-in-use process, rather than merely asked them to describe it. However, it also reflected that the clinicians had opinions about many aspects of the process and therefore mostly talked about it in non-neutral terms.

<table><caption>**Table 3.** Perception of the design-in-use approach to implementation, N = 433 comments.</caption><thead><tr><th rowspan="2">Classification Category</th><th rowspan="2"></th><th colspan="3">Valence</th><th rowspan="2">Total</th><th rowspan="2">Percent</th></tr><tr><th>Pos.</th><th>Neg.</th><th>Neu.</th></tr></thead><tbody><tr><td rowspan="3">Valence</td><td>Positive</td><td>174</td><td>0</td><td>0</td><td>174</td><td>40</td></tr><tr><td>Negative</td><td>0</td><td>209</td><td>0</td><td>209</td><td>48</td></tr><tr><td>Neutral</td><td>0</td><td>0</td><td>50</td><td>50</td><td>12</td></tr><tr><td rowspan="3">Object</td><td>Information technology</td><td>61</td><td>99</td><td>15</td><td>175</td><td>40</td></tr><tr><td>Work process</td><td>24</td><td>38</td><td>15</td><td>77</td><td>18</td></tr><tr><td>Both</td><td>89</td><td>72</td><td>20</td><td>181</td><td>42</td></tr><tr><td rowspan="4">Stage</td><td>Preparing change</td><td>43</td><td>86</td><td>14</td><td>143</td><td>33</td></tr><tr><td>Implementing change</td><td>111</td><td>99</td><td>19</td><td>229</td><td>53</td></tr><tr><td>Training and support</td><td>19</td><td>20</td><td>17</td><td>56</td><td>13</td></tr><tr><td>Other</td><td>1</td><td>4</td><td>0</td><td>5</td><td>1</td></tr><tr><td rowspan="3">Scope</td><td>Intradepartmental</td><td>82</td><td>59</td><td>26</td><td>167</td><td>39</td></tr><tr><td>Interdepartmental</td><td>65</td><td>60</td><td>11</td><td>136</td><td>31</td></tr><tr><td>Both</td><td>27</td><td>90</td><td>13</td><td>130</td><td>30</td></tr></tbody></table>

With respect to the *object* classification it was evident that the clinicians' perception of the design-in-use process revolved around the whiteboard technology: 40% of the comments concerned the technology and another 42% the combination of technology and work processes. Only 18% of the comments were exclusively about work processes. That is, the clinicians tended to think of the changes in terms of the whiteboard because it triggered the changes or made them possible. For example, the physiotherapists' improved possibilities for planning their work was thought of in terms of the decision to use of the whiteboard for ordering physiotherapy because the improved planning possibilities was contingent on this use of the whiteboard.
Notably, the categories of the object classification contained different proportions of negative comments. While 57% of the comments about technology were negative, only 40% of the comments about the combination of technology and work processes were negative. Comments about the technology tended to concern its limitations, while comments about the combination of technology and work processes more often were about how the design-in-use process led to constructive use of the whiteboard.

The *stage* classification showed that the clinicians mostly perceived the design-in-use process as implementing change (53%). Implementing change was the stage at which the design-in-use process influenced the clinical work directly by requiring the clinicians to change their ways of working. That is, implementing change involved the super users, who were driving the change, as well as the end-users, who were to adopt it. In contrast, the stages of preparing change (33%) and training and support (13%) were somewhat removed from the clinical work. As much as 60% (86 of 143) of the comments about preparing change were negative. For example, one end-user stated that “*We do not need more information on the whiteboard*”, thereby indicating that preparing change was superfluous. The proportion of negative comments was lower for implementing change (43%) and for training and support (36%).

With respect to the *scope* classification the design-in-use process was fairly evenly distributed across intradepartmental (39%), interdepartmental (31%), and combined (30%) change. Intradepartmental changes involved fewer clinicians with better opportunities for informally discussing design-in-use ideas because they routinely met during their shifts. Conversely, interdepartmental changes were more organizationally complex to achieve but resulted in some of the most valued changes. For example, the coordination of when patients were ready for surgery became more transparent after the ready-for-surgery checklist was moved to the whiteboard. A super user explained:

> There are seven things that must be satisfied: Is the patient wearing a wristband [with a barcode]? Is the patient fasting? Has a surgeon been down to see the patient? You can follow the progress. And when they have seven out of seven [checklist items ticked off] then they can be picked up. I think it works well.

By moving the checklist from paper to the whiteboard the patients' progress toward satisfying all seven checklist items was no longer just available to the general ward that filled out the checklist but also to the surgical ward that was to receive the patient. This improved the possibilities for planning at the surgical ward. The relative simplicity of intradepartmental change was reflected in a high proportion of positive comments (49%), whereas the higher complexity of the changes that involved both intra- and interdepartmental change resulted in 69% negative comments.

## 4.2 Opportunities and Barriers

The clinicians experienced 15 different opportunities in the design-in-use process and 16 different barriers to it, see Tables 4 and 5. A key finding from the analysis of opportunities and barriers was that the clinicians disagreed substantially. For example, 18 comments described how the design-in-use process gained momentum from early successes and external events (Opportunity 1, Table 4) but, at the same time, 20
comments stated that the design-in-use process lacked momentum (Barrier 4, Table 5). In total, 54% of the 143 opportunity comments (Opportunities 1, 2, 5, 7, 9, 11, 13, 15) were contradicted by 57% of the 224 barrier comments (Barriers 4, 11, 14, 8, 6, 1, 2, 7). That is, about half of the opportunities and barriers were contested by clinicians who held opposing views. This amount of disagreement about the pros and cons of the design-in-use process might constitute a meta-level barrier.

<table><caption>**Table 4.** Opportunities, N = 143.</caption><thead><tr><td>Opportunity group</td><td>Count</td></tr></thead><tbody><tr><td>1. Gaining momentum from early successes and external events</td><td>18</td></tr><tr><td>2. Design-in-use approach allows for meeting local needs</td><td>16</td></tr><tr><td>3. Gradual realization and incorporation of needs and possibilities</td><td>16</td></tr><tr><td>4. Opportunities for better coordination across departments, professional groups etc.</td><td>16</td></tr><tr><td>5. Whiteboard-mediated coordination is an improvement over phone calls</td><td>15</td></tr><tr><td>6. Super users serve as champions</td><td>13</td></tr><tr><td>7. Whiteboard is easy to use, so training and support have not taxed the super users</td><td>12</td></tr><tr><td>8. Opportunities for better overview</td><td>8</td></tr><tr><td>9. Opportunities for standardization</td><td>6</td></tr><tr><td>10. Good balance between benefit and data entry</td><td>6</td></tr><tr><td>11. Design-in-use approach makes for a proactive and engaging process</td><td>5</td></tr><tr><td>12. Exploiting benefits of printing the whiteboard</td><td>5</td></tr><tr><td>13. Exploiting benefits of integration with other systems</td><td>4</td></tr><tr><td>14. Works because nurses and secretaries perform data entry for physicians</td><td>2</td></tr><tr><td>15. Physicians see a point in being part</td><td>1</td></tr></tbody></table>

Four of the opportunities captured what might be considered generic design-in-use qualities: (a) design-in-use approach allowed for meeting local needs, (b) gradual realization and incorporation of needs and possibilities, (c) design-in-use approach made for a proactive and engaging process, and possibly (d) gaining momentum from early successes and external events. In addition, two opportunities emphasized the important facilitating conditions that the super users championed the design-in-use process and that the process achieved a good balance between benefit and data entry. The remaining opportunities were specific to the whiteboard and the local use context, including the opportunities for better coordination, overview, and standardization. Seizing such opportunities was central to the design-in-use process. For example, it was unanticipated that the clinicians began printing the whiteboard to have it at hand at all times and to obtain a copy for personal annotation. A nurse expressed the value of this seemingly unsophisticated change in the use of the whiteboard: "It was at that point I said: Now we are approaching something useful." In one department the accidental feature that the printout extended over two pages became a clear-cut way of assigning responsibility for the patients to the physicians on duty:

> We do it in the way that one [physician] takes the first page of patients – because the printout is on two pages – and the other physician takes the patients on the second page. That helps a lot.
<table><caption>**Table 5.** Barriers, N = 224.</caption><thead><tr><td>Barrier group</td><td>Count</td></tr></thead><tbody><tr><td>1. A misunderstanding to believe that clinicians want to engage in design in use</td><td>23</td></tr><tr><td>2. Lack of system integrations limits whiteboard possibilities and duplicates work</td><td>22</td></tr><tr><td>3. Design-in-use process experienced as rushed, top-down, and administrative</td><td>21</td></tr><tr><td>4. Lack of momentum</td><td>20</td></tr><tr><td>5. Competing with other activities for resources and attention</td><td>17</td></tr><tr><td>6. Lack of standardization restricts the interdepartmental use of the whiteboard</td><td>15</td></tr><tr><td>7. Physicians have been peripheral to the design-in-use process</td><td>14</td></tr><tr><td>8. The need for training and support has been underestimated</td><td>14</td></tr><tr><td>9. Limitations inherent in the whiteboard technology and their physical location</td><td>13</td></tr><tr><td>10. Process has lacked direction, which should have been provided top-down</td><td>12</td></tr><tr><td>11. Little local need for the changes that can be made by design in use of whiteboard</td><td>12</td></tr><tr><td>12. Implementation of work processes lags behind implemented whiteboard facilities</td><td>12</td></tr><tr><td>13. Standardization across departments conflicts with design in use within departments</td><td>9</td></tr><tr><td>14. Replacing phone calls with whiteboard-mediated coordination is not non-loss</td><td>7</td></tr><tr><td>15. Design-in-use approach is dependent on the few committed people</td><td>7</td></tr><tr><td>16. Super users lack required knowledge</td><td>6</td></tr></tbody></table>

The most frequently mentioned barrier was that it was a misunderstanding to believe that the clinicians wanted to engage in design in use. A physician end-user explained that his interest was to treat patients, an interest he considered common to the physicians: “We have our focus on the patients. The IT systems we use are those we have to use. It is not that IT interests us.” From this point of view the design-in-use process was an unwelcome distraction because it took time away from the patients. The second-most frequently mentioned barrier was specific to the whiteboard and the local context. This barrier emphasized the lack of integration between the whiteboard and other clinical systems. Thus, using the whiteboard for new purposes tended to involve duplication of work. The last of the top-three barriers was the experience that the design-in-use process had been rushed, top-down, and oriented toward administrative issues: “They want to extract some numbers, to be able to see how often we tick off this or that, whether we have ticked it off, what we use.” That is, the design-in-use approach had not been presented and performed in a manner that had made this end-user feel part of the process; rather the process was perceived as driven by concerns for quality assessment rather than clinical utility. The remaining barriers included lack of momentum (which meant that too little happened or it happened too slowly), lack of time (because other activities demanded resources and attention), lack of direction (which created uncertainty about what kinds of changes to pursue), lack of knowledge (among the super users who were to drive the design-in-use process), and lack of standardization (which restricted the possibilities for interdepartmental use of the whiteboard). A further barrier with respect to standardization was that the standards that were implemented to facilitate interdepartmental whiteboard use restricted the possibilities for tailoring the whiteboard to intradepartmental needs. These restrictions were extra frustrating because the need for standardization was only
realized gradually; thus, some early intradepartmental changes had to be rolled back to comply with standards that were introduced later to facilitate interdepartmental use:

*We end up doing everything twice. It was hugely frustrating that we had set it all up the way we wanted – we thought that now it is perfect for me – only to realize that this doesn't work [interdepartmentally]. That's a bummer. Then we had to change a lot of it. And that is worse than the initial sense of freedom.*

## 4.3 Variation Across User Groups

To investigate variation across user groups we selected the roles and professional groups with at least three representatives among the interviewees. For roles this selection led to comparing the ten super users with the five end-users; for professional groups it led to comparing the six physicians with the six nurses. Table 6 shows the results. Overall, the comments were somewhat differently distributed for super users versus end-users and similarly distributed for physicians versus nurses.

<table><caption>**Table 6.** Variation across user groups.</caption><thead><tr><th rowspan="2">Classification Category</th><th colspan="2">Role</th><th colspan="2">Professional group</th></tr><tr><th>Super user %</th><th>End-user %</th><th>Physician %</th><th>Nurse %</th></tr></thead><tbody><tr><td>Valence</td><td></td><td></td><td></td><td></td></tr><tr><td>Positive</td><td>48</td><td>30</td><td>33</td><td>40</td></tr><tr><td>Negative</td><td>38</td><td>56</td><td>57</td><td>48</td></tr><tr><td>Neutral</td><td>14</td><td>13</td><td>10</td><td>12</td></tr><tr><td>Object</td><td></td><td></td><td></td><td></td></tr><tr><td>Information technology</td><td>39</td><td>36</td><td>39</td><td>47</td></tr><tr><td>Work process</td><td>14</td><td>23</td><td>21</td><td>12</td></tr><tr><td>Both</td><td>47</td><td>42</td><td>40</td><td>40</td></tr><tr><td>Stage</td><td></td><td></td><td></td><td></td></tr><tr><td>Preparing change</td><td>31</td><td>30</td><td>38</td><td>32</td></tr><tr><td>Implementing change</td><td>54</td><td>54</td><td>47</td><td>52</td></tr><tr><td>Training and support</td><td>14</td><td>13</td><td>11</td><td>15</td></tr><tr><td>Other</td><td>1</td><td>3</td><td>4</td><td>0</td></tr><tr><td>Scope</td><td></td><td></td><td></td><td></td></tr><tr><td>Intradepartmental</td><td>39</td><td>25</td><td>34</td><td>41</td></tr><tr><td>Interdepartmental</td><td>25</td><td>39</td><td>28</td><td>33</td></tr><tr><td>Both</td><td>36</td><td>36</td><td>39</td><td>25</td></tr></tbody></table>

Two differences should be noted. First, the super users were more positive and less negative than the end-users (see Table 6). Either the clinicians who were more positive about the design-in-use approach became super users or the super users' larger involvement in this process made them more positive. This difference underlined the barrier that the design-in-use process was dependent on the few clinicians who were committed to the process (Barrier 15, Table 5). These few
committed clinicians were in the group of super users. Second, the super users focused more on intradepartmental change, whereas the end-users focused more on interdepartmental change (see Table 6). Intradepartmental change was the super users' immediate responsibility and it was, probably, easier for them to accomplish because they had their primary knowledge and network in their department. In contrast, interdepartmental change involved negotiation and alignment with other departments. While the super users as a group appeared somewhat reluctant to engage this increased complexity, the end-users focused on the interdepartmental changes because they were the most evident outcomes of the design-in-use process. Several end-users had some difficulty giving examples of outcomes other than the interdepartmental changes of using the whiteboard for ordering physiotherapy and for coordinating when patients were ready for surgery.

# 5 Discussion

Design in use will only happen if motivated users make it happen. Thus, the clinicians' perception of the design-in-use approach is both an outcome of the implementation process and a key input to it. If they perceive the approach negatively, it is unlikely to succeed. If it succeeds, they are likely to perceive it positively.

## 5.1 How Do Users Perceive a Design-in-Use Approach to Implementation?

At the studied hospital the clinicians have one and a half years of experience with the design-in-use approach to the implementation of the whiteboard. That is, their perception of the approach has had time to form and settle. In summary, we find that:

* The clinicians perceive the design-in-use approach to implementation in conflicting ways. In addition to many positive as well as many negative comments, the clinicians disagree about the associated opportunities and barriers.

* The super users are more positive about the approach than the end-users. As a consequence the super users serve as champions for the system, and the design-in-use approach is highly dependent on these few, committed people.

* Intradepartmental change is perceived more positively, probably because it is easier to achieve. The super users focus more on intradepartmental change, the end-users more on interdepartmental change, which is possibly more valuable.

* Standardization across departments conflicts with design in use within departments. While the clinicians acknowledge the need for standardization, the conflict affects their perception of the design-in-use approach negatively.

* The design-in-use approach is inextricably sociotechnical but the clinicians perceive the technology – the whiteboard – as more prominent than the work process. Less than one in five comments are exclusively about the work process.

* The clinicians' perception of design in use is more about implementing change than about preparing it or about training and support. That is, the approach becomes salient to the clinicians when its results start to influence their work.
* While the super users and end-users perceive the design-in-use approach somewhat differently, the physicians and nurses perceive it similarly. Thus, the clinicians' perception of the approach varies with their role in it, not with their primary task.

The clinicians' conflicting perceptions of the design-in-use approach have multiple and interrelated sources. One of these sources is the scope of the design-in-use activities. We propose that the clinicians probably perceive intradepartmental change more positively because it is easier to achieve and therefore proceeds more smoothly. In contrast, interdepartmental change is more organizationally complex, extends beyond the super users' immediate network, and requires the development and adoption of standards that reduce the clinicians' freedom in configuring the whiteboard for intradepartmental needs. This argument accords with Sanchez and Mahoney [37], who emphasize that changes internal to organizational components are much simpler than changes that cut across component boundaries because intra-component changes can be handled locally whereas inter-component changes require organization-wide learning and decisions. That said, the potential benefit of inter-component changes is larger because they may introduce structural improvements. The ordering of physiotherapy provides an example. However, the standardization involved in pursuing interdepartmental changes is a source of frustration and exemplifies the generally recognized friction between global and local concerns in the evolution of an infrastructure [40]. While this study shows that the clinicians' appreciation of design in use interrelates with the scope of the pursued changes, further work is needed to spell out these interrelations in detail.

Another source of the clinicians' perceptions of the design-in-use approach is its blending of technical and work-process change. The technology — the whiteboard — features prominently in the clinicians' perception of the design-in-use approach but in contrast to Mehandjiev et al. [22] without worrying about the technical quality of the configurations. The absence of this worry may suggest that the clinicians are overconfident in the quality of their configurations [17] or that it is simple to configure the whiteboard. In either case, the perceived simplicity of configuring the whiteboard is an important contextual factor because it means that the super users have more time available for revising the work processes associated with the whiteboard. The design-in-use process is at least as much about revising work processes as it is about technical configuration. This sociotechnical outlook probably reflects that the whiteboard as such is of negligible interest to the clinicians, who instead perceive the whiteboard as the driver of changes that improve their work processes or, in other clinicians' opinion, largely fail to improve them. It reinforces the sociotechnical outlook that the design-in-use activities occur during use; traditional design (i.e., design before use) may be more prone to a predominantly technical outlook. The qualities of design in use, as opposed to design before use, are also highlighted by the finding that the clinicians' perception of the design-in-use approach is more about implementing change than about preparing it or about training and support. Truex et al. [43] argue that it is when a design starts to affect users in their daily work that the design becomes salient to them and they become motivated to influence it. That is, the clinicians are more likely to realize the implications of the
whiteboard after they have started to use it and they are more likely to be able and motivated to voice their needs and concerns after they have started to use it.

A further source of the clinicians' perceptions of the design-in-use approach is whether they expect to receive a finalized system or are prepared to engage in design in use. Many clinicians expect and prefer to receive a system that has already been configured for their needs and, thus, is ready for use. However, some clinicians are prepared to continue the design of the system and associated work practices on the basis of their experiences from starting to use the system. The latter group appears to engage in design in use because they like it, find it useful, and believe they will be good at it [6]. When previous design-in-use studies distinguish between different groups of people it has mostly been to investigate collaborations and tensions between users and developers [8], between users and management [41], and between super users and technical IT support [46]. It has not been to investigate how different user groups may disagree about whether design in use is an appealing process.

We find that the distinction between super users and end-users explains some of the difference in the clinicians' perception of the design-in-use process but that the distinction between physicians and nurses does not. The clinicians' conflicting perceptions of the design-in-use approach makes it a critical decision who are selected as super users to drive the process but it also adds to the super users' task by extending it with a role of championing the whiteboard. To champion the whiteboard the super users must be able to influence their colleagues' attitudes and behavior. They must also be prepared to employ an outgoing and advocating approach rather than merely to provide opportunities that their colleagues may adopt or bypass as they see fit. Management could have supported the super users in this championing role but, instead, adopted a rather hands-off approach, which made it easier for the disinclined clinicians to remain uncommitted to the design-in-use process.

## 5.2 Implications

We see four implications of the study for a design-in-use approach to implementation. First, a design-in-use approach to implementation is dependent on a limited number of positively inclined users. It is unadvisable to adopt a design-in-use approach unless such users can be identified ahead of the implementation process. These users cannot be presumed to have all the competences necessary to accomplish design in use. Targeted support is necessary. Second, design-in-use activities with a wide scope are perceived more negatively. Thus, a design-in-use approach may be best suited for intradepartmental change. While appreciated intradepartmental change can be accomplished through design-in-use processes that proceed bottom-up, interdepartmental change requires more coordination, more formality, and probably a collaboration more similar to mutual development [24]. Third, some users expect to receive a finalized system. These users will experience a design-in-use approach to implementation as unprofessional and they will be reluctant to take time away from their primary tasks to contribute to design-in-use activities. Organizations can work to change their attitude, can task those who engage in design-in-use activities with the additional task of championing the system, or can risk underutilizing the system.
Fourth, in order for design in use to happen the system must be used. Thus, to get the design-in-use approach going management must ensure system use. This requires clarity about the purposes for which the system must, as a minimum, be used.

## 5.3 Limitations

Three limitations should be remembered in interpreting the results of this study. First, the study is restricted to one hospital and one design-in-use process. The results cannot be presumed to generalize to all design-in-use approaches to implementation. On the contrary, design in use is a situated process through which a particular system and a particular use context are adapted to each other. Second, while the interviewees span six professional groups, we acknowledge that the majority of the interviewees are physicians or nurses. For example, there is only one secretary among the interviewees even though secretaries are involved in the clinical work [23] and in the use of the studied whiteboard. In addition, we interviewed more super users than end-users but at the hospital there are many more end-users than super users. The rationale for including many super users in the sample was their central role in the design-in-use activities. Third, our sample of interviewees is too small to enable statistical analyses of whether the users' perception of the design-in-use approach differs across professional groups or between super users and end-users. We would welcome a large-scale survey of how different user groups perceive design-in-use processes. The present study provides categories and findings for informing such a survey.

# 6 Conclusion

The clinicians at the studied hospital hold conflicting views about the design-in-use approach to the implementation of the network of interconnected electronic whiteboards. While some clinicians, especially the super users, welcome the approach and find that it allows for meeting gradually realized local needs, other clinicians expect new systems to be fully configured prior to go-live. The conflicting perceptions show that a design-in-use approach to implementation is not a panacea. Rather, it requires careful communication, targeted support, and may be better suited for intradepartmental than interdepartmental change.

## Acknowledgements

This paper is dedicated to the memory of the second author, Arnvør Torkilsheyggi, who tragically died before the paper saw publication. The paper reports work conducted in the Clinical Communication project, which was a research and development collaboration between Region Zealand, Imatis, Roskilde University, and University of Copenhagen. The second author's participation in the project was co-funded by Region Zealand. We are grateful to Claus R. Mortensen for his support in identifying the interviewees. Special thanks are due to the interviewees.
## References

1. Aanestad, M., Driveklepp, A.M., Sørli, H., Hertzum, M.: Participatory continuing design: "Living with" videoconferencing in rehabilitation. In: Kanstrup, A.M., Bygholm, A., Bertelsen, P., Nøhr, C. (eds.) *Participatory Design and Health Information Technology*, pp. 45-59. IOS Press, Amsterdam (2017).

2. Aarts, J., Berg, M.: Same system, different outcomes: Comparing the implementation of computerized physician order entry in two Dutch hospitals. *Methods of Information in Medicine* 45(1), 53-61 (2006).

3. Balka, E., Wagner, I.: Making things work: Dimensions of configurability as appropriation work. In: *Proceedings of the CSCW 2006 Conference on Computer Supported Cooperative Work*, pp. 229-238. ACM Press, New York (2006).

4. Bikson, T.K., Eveland, J.D.: Sociotechnical reinvention: Implementation dynamics and collaboration tools. *Information, Communication & Society* 1(3), 270-290 (1998).

5. Bjögvinsson, E., Ehn, P., Hillgren, P.-A.: Design things and design thinking: Contemporary participatory design challenges. *Design Issues* 28(3), 101-116 (2012).

6. Blackwell, A.F.: End-user developers - What are they like? In: Paternó, F., Wulf, V. (eds.) *New Perspectives in End-User Development*, pp. 121-135. Springer, Cham (2017).

7. Bowers, J.: The work to make a network work: Studying CSCW in action. In: *Proceedings of the CSCW'94 Conference on Computer Supported Cooperative Work*, pp. 287-298. ACM Press, New York (1994).

8. Dittrich, Y., Bolmsten, J., Eriksson, J.: End user development and infrastructuring - Sustaining organizational innovation capabilities. In: Paternó, F., Wulf, V. (eds.) *New Perspectives in End-User Development*, pp. 165-206. Springer, Cham (2017).

9. Dittrich, Y., Eriksén, S., Hansson, C.: PD in the wild: Evolving practices of design in use. In: Binder, T., Gregory, J., Wagner, I. (eds.) *PDC2002: Proceedings of the Seventh Conference on Participatory Design*, pp. 124-134. CPSR, Palo Alto, CA (2002).

10. Fischer, G.: End-user development and meta-design: Foundations for cultures of participation. *Journal of Organizational and End User Computing* 22(1), 52-82 (2010).

11. Hartswood, M., Proctor, R., Slack, R., Voss, A., Büscher, M., Rouncefield, M., Rouchy, P.: Co-realisation: Towards a principled synthesis of ethnomethodology and participatory design. *Scandinavian Journal of Information Systems* 14(2), 9-30 (2002).

12. Henderson, A., Kyng, M.: There's no place like home: Continuing design in use. In: Greenbaum, J., Kyng, M. (eds.) *Design at Work: Cooperative Design of Computer Systems*, pp. 219-240. Erlbaum, Hillsdale, NJ (1991).

13. Hertzum, M.: Electronic emergency-department whiteboards: A study of clinicians' expectations and experiences. *International Journal of Medical Informatics* 80(9), 618-630 (2011).

14. Hertzum, M., Simonsen, J.: Effects-driven IT development: Specifying, realizing, and assessing usage effects. *Scandinavian Journal of Information Systems* 23(1), 3-28 (2011).

15. Hertzum, M., Simonsen, J.: Configuring information systems and work practices for each other: What competences are needed locally? *International Journal of Human-Computer Studies* 122, 242-255 (2019).
16. Karasti, H., Baker, K., Millerand, F.: Infrastructure time: Long-term matters in collaborative development. *Computer Supported Cooperative Work* 19(3&4), 377-415 (2010).

17. Ko, A.J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrence, J., Lieberman, H., Myers, B., Rosson, M.B., Rothermel, G., Shaw, M., Wiedenbeck, S.: The state of the art in end-user software engineering. *ACM Computing Surveys* 43(3), article 21 (2011).

18. Ludwig, T., Dax, J., Pipek, V., Wulf, V.: A practice-oriented paradigm for end-user development. In: Paternó, F., Wulf, V. (eds.) *New Perspectives in End-User Development*, pp. 23-41. Springer, Cham (2017).

19. Maceli, M., Atwood, M.E.: "Human crafters" once again: Supporting users as designers in continuous co-design. In: Dittrich, Y., Burnett, M., Mørch, A., Redmiles, D. (eds.) *Proceedings of the IS-EUD 2013 International Symposium on End User Development*, Vol. LNCS 7897, pp. 9-24. Springer, Berlin (2013).

20. Mackay, W.E.: Patterns of sharing customizable software. In: *Proceedings of the CSCW'90 Conference on Computer Supported Cooperative Work*, pp. 209-221. ACM Press, New York (1990).

21. Markus, M.L.: Technochange management: Using IT to drive organizational change. *Journal of Information Technology* 19(1), 4-20 (2004).

22. Mehandjiev, N., Sutcliffe, A., Lee, D.: Organizational view of end-user development. In: Lieberman, H., Paternó, F., Wulf, V. (eds.) *End User Development*, pp. 371-399. Springer, Dordrecht (2006).

23. Møller, N.L.H., Vikkelsø, S.: The clinical work of secretaries: Exploring the intersection of administrative and clinical work in the diagnosing process. In: Dugdale, J., Masclet, C., Grasso, M., Boujut, J.F., Hassanaly, P. (eds.) *From Research to Practice in the Design of Cooperative Systems: Results and Open Challenges*, pp. 33-47. Springer, London (2012).

24. Mørch, A.I., Andersen, R.: Mutual development: The software engineering context of end-user development. *Journal of Organizational and End User Computing* 22(2), 36-57 (2010).

25. Nah, F.F.-H., Zuckweiler, K.M., Lau, J.L.-S.: ERP implementation: Chief information officers' perceptions of critical success factors. *International Journal of Human-Computer Interaction* 16(1), 5-22 (2003).

26. Nicolaou, A.I.: Firm performance effects in relation to the implementation and use of enterprise resource planning systems. *Journal of Information Systems* 18(2), 79-105 (2004).

27. Okamura, K., Fujimoto, M., Orlikowski, W.J., Yates, J.: Helping CSCW applications succeed: The role of mediators in the context of use. *The Information Society* 11(3), 157-172 (1995).

28. Orlikowski, W.J.: Improvising organizational transformation over time: A situated change perspective. *Information Systems Research* 7(1), 63-92 (1996).

29. Orlikowski, W.J., Hofman, J.D.: An improvisational model for change management: The case of groupware technologies. *Sloan Management Review* 38(2), 11-22 (1997).

30. Park, S.Y., Chen, Y., Rudkin, S.: Technological and organizational adaptation of EMR implementation in an emergency department. *ACM Transactions on Computer-Human Interaction* 22(1), 1:01-1:24 (2015).
31. Pipek, V., Kahler, H.: Supporting collaborative tailoring. In: Lieberman, H., Paternó, F., Wulf, V. (eds.) *End User Development*, pp. 315-345. Springer, Dordrecht (2006).

32. Pipek, V., Wulf, V.: A groupware's life. In: Bødker, S., Kyng, M., Schmidt, K. (eds.) *ECSCW'99: Proceedings of the Sixth European Conference on Computer-Supported Cooperative Work*, pp. 199-218. Kluwer, Amsterdam (1999).

33. Powell, S.G., Baker, K.R., Lawson, B.: A critical review of the literature on spreadsheet errors. *Decision Support Systems* 46(1), 128-138 (2008).

34. Pries-Heje, L., Dittrich, Y.: ERP implementation as design: Looking at participatory design for means to facilitate knowledge integration. *Scandinavian Journal of Information Systems* 21(2), 27-58 (2009).

35. Priestman, W., Sridharan, S., Vigne, H., Collins, R., Seamer, L., Sebire, N.J.: What to expect from electronic patient record system implementation: Lessons learned from published evidence. *Journal of Innovation in Health Informatics* 25(2), 92-104 (2018).

36. Rasmussen, R., Fleron, B., Hertzum, M., Simonsen, J.: Balancing tradition and transcendence in the implementation of emergency-department electronic whiteboards. In: Molka-Danielsen, J., Nicolaisen, H.W., Persson, J.S. (eds.) *Selected Papers of the Information Systems Research Seminar in Scandinavia 2010*, pp. 73-87. Tapir Academic Press, Trondheim, NO (2010).

37. Sanchez, R., Mahoney, J.T.: Modularity, flexibility, and knowledge management in product and organization design. *Strategic Management Journal* 17(Winter Special Issue), 63-76 (1996).

38. Simonsen, J., Hertzum, M.: Sustained participatory design: Extending the iterative approach. *Design Issues* 28(3), 10-21 (2012).

39. Spencer, M.: Brittleness and bureaucracy: Software as a material for science. *Perspectives on Science* 23(4), 466-484 (2015).

40. Star, S.L., Ruhleder, K.: Steps toward an ecology of infrastructure: Design and access for large information spaces. *Information Systems Research* 7(1), 111-134 (1996).

41. Torkilsheyggi, A., Hertzum, M.: Incomplete by design: A study of a design-in-use approach to systems implementation. *Scandinavian Journal of Information Systems* 29(2), article 2 (2017).

42. Trigg, R.H., Bødker, S.: From implementation to design: Tailoring and the emergence of systematization in CSCW. In: *Proceedings of the CSCW'94 Conference on Computer Supported Cooperative Work*, pp. 45-54. ACM Press, New York (1994).

43. Truex, D.P., Baskerville, R., Klein, H.: Growing systems in emergent organizations. *Communications of the ACM* 42(8), 117-123 (1999).

44. Tyre, M.J., Orlikowski, W.J.: Windows of opportunity: Temporal patterns of technological adaptation in organizations. *Organization Science* 5(1), 98-118 (1994).

45. Yetim, F., Draxler, S., Stevens, G., Wulf, V.: Fostering continuous user participation by embedding a communication support tool in user interfaces. *AIS Transactions on Human-Computer Interaction* 4(2), 153-168 (2012).

46. Åsand, H.-R.H., Mørch, A.I.: Super users and local developers: The organization of end user development in an accounting company. *Journal of Organizational and End User Computing* 18(4), 1-21 (2008).