# Designing and Evaluating Dashboards for Multi-agency Crisis Preparation: A Living Lab

Nitesh Bharosa, Marijn Janssen, Sebastiaan Meijer, Fritjof Brave

► To cite this version:

Nitesh Bharosa, Marijn Janssen, Sebastiaan Meijer, Fritjof Brave. Designing and Evaluating Dashboards for Multi-agency Crisis Preparation: A Living Lab. 9th IFIP WG 8.5 International Conference on Electronic Government (EGOV), Aug 2010, Lausanne, Switzerland. pp.180-191, 10.1007/978-3-642-14799-9_16. hal-01056587

HAL Id: hal-01056587

https://inria.hal.science/hal-01056587

Submitted on 20 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Designing and evaluating dashboards for multi-agency crisis preparation: a Living Lab

Nitesh Bharosa<fnref n="1" />, Marijn Janssen<fnref n="1" />, Sebastiaan Meijer<fnref n="1" />, Fritjof Brave<fnref n="2" />

**Abstract.** Public organizations show growing interest in the development of dashboards that aid relief agency managers in crisis preparation. Yet, there is a dearth of research on the development of such dashboards. This paper discusses the experiences gained from a pioneering Living Lab on the development and evaluation of dashboards for assessing crisis preparedness. In order to evaluate and further improve dashboards, a two-day user-centered gaming simulation was organized with forty relief agency managers. A survey distributed amongst the managers indicates that they were satisfied with the dashboards and intend to use these in practice. However, the managers suggested that the formulation and clustering of the performance indicators requires better alignment with the context of use. One of the main findings is that the high level of uncertainty regarding the final set of performance indicators and the corresponding norms demands flexibility in the dashboard architecture beyond the evaluation stage.

**Keywords:** Dashboards, Living Lab, crisis preparation, crises management, gaming, simulation, IS success

## 1 Introduction

In e-government disparate public agencies have to coordinate their activities with each other horizontally and vertically [1]. Crises preparation and response are a subset of e-government, in which public organizations (i.e., police, fire department and ambulance services) need to coordinate their activities in real-time [2]. As the occurrence and evolution of a crisis cannot be predicted in advance, it is of vital importance to be prepared in order to enable rapid crisis response. This has resulted in an increasing interest in crisis preparedness of the main relief agencies, especially since some of the major crises in the past decade (e.g., 9/11, Katrina, London, Madrid) have exhibited poor crisis preparation. Due to the impact and associated media attention, policy makers cannot afford to say “we were unprepared” anymore to victims and their families in case of a crisis [3]. Hence, relief agency managers are expected to prepare for the eventuality of a crisis by understanding the vulnerabilities

<footnote n="1">¹Delft University of Technology, Jaffalaan 5, 2628 BX Delft, The Netherlands
{n.bharosa; m.f.w.h.a.janssen; s.meijer}@tudelft.nl</footnote>

<footnote n="2">²Berenschot Groep B.V., Europalaan 40, 3526 KS Utrecht, The Netherlands

f.brave@berenschot.nl</footnote>
of an organization, analyzing the organizational capability to deal with a range of
crisis scenarios, and by taking precautionary measures to mitigate the possible risks of
being unable to cope with crisis events. In each of these crisis preparation processes,
performance indicators (PIs) are considered of major importance [4]. Historically,
relief agencies operate in a silo-ed manner and define and use their own set of PIs.
They usually focus their PIs on internal processes, clustered in themes such as
financial status, human resources, and service delivery.

In general, relief agency managers depend on governmental agencies for their
financial resources. Since policy makers usually have a fixed budget for relief
agencies, they need to know how to balance financial resources between agencies in
order to maintain an overall level of preparedness. For policymakers, PIs are essential
for planning crisis preparedness. Yet the current mono-agency sets of PIs do not show
the aggregate level of preparedness of the relief services as a whole, which in turn is
the criterion by which the public will judge governmental agencies.

Scholars in the domains of strategic management [e.g., 5, 6] have proposed the use
of dashboards as instruments for both the clustering and visualization of PIs. A
dashboard is “a visual display of the most important information needed to achieve
one or more objectives, consolidated and arranged on a single screen so the
information can be monitored at a glance” [7, p. 34]. Despite the advantages predicted
for organizations when using dashboards [e.g., 8, 9-10], literature on the development
of dashboards indicating the level of crisis preparedness on a multi-agency scale is
scarce. Instead, most studies are concerned with the appropriateness or success of
response activities. Accordingly, the objective of this paper is to present experiences
extracted from the development and evaluation of dashboards in practice. The authors
pursued this objective by employing a Living Lab approach, in which academics,
relief agency managers, and policy makers join forces in order to achieve a common
purpose. This paper contributes to existing literature on crisis preparation by
presenting experiences extracted from dashboard development and evaluation. In
addition, this paper elaborates on the types of dashboard required for crisis
preparation in a multi-agency environment.

The next section presents theoretical backgrounds of the dashboard concept. Then, we discuss the Living Lab on dashboard development in The Netherlands, followed by a brief description of the resulting dashboards. Here, we explicitly focus on the design choices and tradeoffs made in this project. Section 4 discusses the setup and results of the dashboard evaluation process, followed by some derived guidelines for developing dashboards for multi-agency crisis preparation. The paper concludes with some conclusions, discussions, and opportunities for further research.

2 A Living Lab for disaster preparation

2.1 Background

At the start of 2008, the Dutch parliament finally passed a long debated law
mandating the formation of twenty-five multi-agency safety organizations. According
to this law, the multi-agency safety organizations that were to be formed would act as
the main responsible entities when it comes to crisis preparation and response in the
geographic region they covered. This meant that previously autonomous relief
agencies, including the police, fire and ambulance services, were now required to collaborate in terms of crisis preparation and response. The law also mandates that the multi-agency safety organizations, needed to conduct crisis preparation activities based on standardized and comparable PIs. In order to comply with the law, five out of twenty-five multi-agency safety organizations went on and agreed to collaborate in the development and use of PIs. The collaboration project was titled ‘Aristoteles’ (after the Greek Philosopher) and started in August 2008. The main goal of the Aristoteles project was to bring together academia and practitioners in the development and evaluation of dashboards visualizing the state of crisis preparedness in the multi-agency safety organization. Since this project was the first of its kind in the Dutch context, a major part of this project required the collaboration of academia and practitioners. The authors were key members of the project group that decided to follow a Living Lab approach. The next subsection discusses and motivates the choice for this approach.

## 2.2 A Living Lab approach to developing dashboards

Both researchers and practitioners show increasing interest in the Living Labs approach to innovation and research in complex design environments involving many stakeholders. Yet, this approach is still relatively new, therefore lacking standard and universally agreed upon definitions and instruments. Pallot [11] argues that a Living Lab is neither a traditional research lab nor a “testbed”, but rather an "innovation platform" that engages all stakeholders such as end-users, researchers, industrialists, policy makers, and so on at the earlier stage of the innovation process. As such, Living Labs allow stakeholders to experiment with breakthrough concepts and assess the potential value for both the society (citizens) and users that will lead to breakthrough innovations. Lama and Origin [12] describe Living Labs as “a user-centric research methodology for sensing, prototyping, validating and refining complex solutions in multiple and evolving real life contexts.” Følstad [13] explained that literature on Living Lab has served to identify two characteristics that discriminate Living Labs from other approaches: (1) Contextualized co-creation: Living Labs supporting context research and co-creation with users, and (2) Testbed association: Living Labs serving as a testbed extension, where testbed applications are accessed in contexts familiar to the users. Living Labs are mostly established through collaboration of private as well as public research partners and can be used with multiple iterations throughout multiple stages of the innovation [14].

The Aristoteles project team decided to employ a Living Lab approach for two main reasons. Firstly, since there were no comparable dashboards for multi-agency disaster preparation in practice, little was known about the specific set of PIs and corresponding dashboards required for the various relief agencies. The project team was convinced that user co-creation, one of the characteristics of Living Labs, would be the most efficient, and yet most effective way to determine the necessary PIs and dashboards. Living Labs can be cost-effective as they avoid making costly changes at a later innovation stage [12, 13]. They also generate better ideas and allow the detection and elimination of the “probably unsuccessful” ideas faster [13]. However, one of the main weaknesses of Living Labs is that they require a lot of time and budget. The second reason for selecting a Living Lab approach is that Living Labs
allow for the combination of quantitative and qualitative research methods for data
collection. As a Living Lab, the Aristoteles project creates a unique opportunity for
researchers to investigate how dashboards can be developed and evaluated in a multi-
agency and inter-regional setting. Figure 1 outlines the main phases of the Living Lab.

<figure><img src="image_2.png" /><figcaption>**Figure 1.** Phases of the Aristoteles Living Lab</figcaption></figure>

The first phase included semi structured interviews with stakeholders, allowing us to generate a long-list of required PIs. The results of this phase include a spreadsheet with over 500 different PIs gathered from the interviews. After completing phase 1, phase 2 and 3 were implemented in parallel. Having a first idea on the type and categories of PIs that needed to be visualized in the dashboards, the team developed the dashboard alternatives. In the meantime, work was done on reducing the initial long-list of PIs gathered from the interviews. We specially pursued a shortlist of PIs with a specific and concise set of PIs we could visualize in the dashboards. In order to make sure no crucial performance indicators were left out in the shortlist, the stakeholders participated in five brainstorms and voting sessions. Each session included a dozen relief agency managers responsible for crisis preparation for their respective agency. The goal of these sessions was to stimulate the actual users of PIs to prioritize the main PIs they needed for the process of multi-agency disaster preparation. The next sections discuss the resulting dashboard prototypes.

3 Dashboard prototypes

Dashboards can be designed and tailored to many specific purposes depending on the task to be supported, the context of use and the frequency of use [7]. Moreover, the various data and purposes that dashboards can be used for are worth distinguishing, as they can demand differences in visual design and functionality. The factor that relates most directly to a dashboard's visual design involves the role it plays, whether strategic, tactical, or operational. The design characteristics of the dashboard can be tailored to effectively support the needs of each of these roles. In line with Morrissey [15], our process of tailoring dashboard content consisted of three phases: (1) identifying the main stakeholders; (2) identifying goals and establishing baseline
capability for each stakeholder; and (3) selecting strategic, tactical, or operational dashboard content aligned with these goals. While certain differences such as these will affect design, there are also many commonalities that span all dashboards and invite a standard set of design practices. Based on the number of relief agencies and the three levels (strategic, tactical, and operational) that needed to be supported, seven different dashboards were developed. Each dashboard display was adaptable from detailed information (tables, trends) to a more abstract level (traffic lights and speedometer). Table 1 summarizes three types of dashboards we developed: strategic, tactical, and operational level dashboards.

<table><caption>**Table 1.** Overview of the developed dashboards and targeted users</caption><thead><tr><td>Dashboard</td><td>User</td><td>User roles</td><td>Dashboard type</td></tr></thead><tbody><tr><td>1</td><td>Fire Department</td><td>Second in command, Financial advisor, Human resource advisor</td><td>Operational, focused on fire department operations</td></tr><tr><td>2</td><td>Ambulance services</td><td>Second in command, Financial advisor, Human resource advisor</td><td>Operational, focused on ambulance services operations</td></tr><tr><td>3</td><td>Emergency control room</td><td>Second in command, Financial advisor, Police department representative</td><td>Operational, focused on the multi-agency performance of the control room</td></tr><tr><td>4</td><td>Crisis management agency</td><td>Director of regional crisis management department, Regional Hazard/risk advisor, Human resource advisor</td><td>Tactical, focused on the multi-agency performance regarding crisis management</td></tr><tr><td>5</td><td>Financial board</td><td>Director of financial department, Financial advisor, Human resource advisor</td><td>Tactical, focused on mid-term financial performance of the multi-agency safety region</td></tr><tr><td>6</td><td>Board of Commanders</td><td>Commanders of the respective relief agencies (five in total)</td><td>Tactical, focused on mid-term overall performance of the multi-agency safety region</td></tr><tr><td>7</td><td>Board of Mayors</td><td>Mayors of the respective municipalities (five in total)</td><td>Strategic, focused on long-term overall performance of the multi-agency safety region</td></tr></tbody></table>

Table 1 shows that we developed three dashboards for the operational level of the multi-agency safety organization. Each type of dashboard serves a different level and user group with different information needs. For the daily crisis preparation process, the team decided that the absolute values and thresholds per PI, based on averages and norms were more important than trends. The dashboards for the operational level are complementary to each other since they display different sets of PIs. The focus of the operational dashboards is daily use in the crisis preparation process. These dashboards are agency specific in scope and therefore tailored to the core processes of the individual relief agencies. The three dashboards developed for the tactical level of the multi-agency safety organization measure short-term (monthly) trends and progress toward strategic initiatives or specific projects. The audience for these dashboards consists of the directors or commanding officers of the relief agencies. Similar to the operational level dashboards, the tactical level dashboards display detailed PIs that relief agency managers need for performing their daily tasks. The tactical dashboards take advantage of awareness of context and the sophistication of relief agency managers to present significantly more detail without sacrificing comprehension. The
emphasis is on highlighting opportunities or identifying risks regarding crisis preparation.

The third type of dashboard was developed for the strategic level stakeholders in the safety region. The following screenshot illustrates this dashboard.

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Screenshot of a dashboard (strategic level)</figcaption></figure>

The strategic level dashboard developed displays aggregated and periodical PIs. The reasoning behind this is that it would be unusual for a top-level manager to use an operational dashboard. The audience for the strategic dashboard consists of the Mayors of municipalities included in the regional multi-agency safety organization. For this audience, graphics summarizing long-term trends are more appropriate than measure showing the day-to-day processes in near real time with the aim of intervening quickly to resolve issues or take advantage of opportunities. The strategic level dashboard was highly summarized, graphical, and less frequently updated since the PIs values represented contained information aggregated over longer periods of time (i.e., yearly values). On this level of crisis preparation, the project team considered the overall performance of the multi-agency safety organization and the trends to be more important than the daily/absolute value of the PIs. Due to the longer time intervals compared to tactical and operational dashboards, the strategic level dashboard was based on various equations and functions that combine the values individual PIs. The strategic dashboards developed also included national, external, trend, and growth measures relevant for the safety region as a whole.
# 4 Dashboard evaluation

## 4.1 Gaming simulation

The fourth phase of the Living Lab included a two-day gaming simulation with forty relief agency managers. Gaming simulation is an approach often applied for awareness creation and learning in strategic management and policy formulation [16]. However, as demonstrated by Meijer et al. [17] gaming simulation is also very instrumental when it comes to the evaluation of artifacts in semi-realistic environments. The gaming simulation served three purposes. The first and most important purpose was to evaluate the preliminary dashboards in a semi-realistic setting. The second purpose was to extract aspects of the dashboards that required further improvement. Finally, the gaming simulation was also a way to demonstrate the results of the Living Lab to the future users (i.e., relief agency managers) and politicians. The following table outlines the gaming simulation activities.

<table><caption>**Table 2.** Overview gaming simulation activities</caption><thead><tr><th></th><th>Period</th><th>Main activities</th></tr></thead><tbody><tr><td rowspan="3">Day 1</td><td>Morning</td><td>-Introduction to the game (purpose, design etc)<br>-Explanation of the dashboards (types, PIs, buttons etc)<br>-Round 1: prepare a crisis plan for 2010<br>-Plenary evaluation of round 1 (focus group)</td></tr><tr><td>Afternoon</td><td>-Round 2: prepare a crisis plan for 2011<br>-Plenary evaluation of round 2 (focus group): what needs to be changed to the dashboards for more efficient and effective crisis preparation?</td></tr><tr><td>Evening</td><td>-Implementation of the changes suggested by the participants in the dashboards (only by the architects)</td></tr><tr><td rowspan="2">Day 2</td><td>Morning</td><td>-Round 3: prepare a crisis plan for 2013<br>-Plenary evaluation of round 3 (focus group)</td></tr><tr><td>Afternoon</td><td>-Plenary evaluation of the entire game<br>-Fill in the questionnaires</td></tr></tbody></table>

The participants were separated in seven teams each using a different dashboard (see Table 1). Accordingly, the relief agency managers were required to engage in several crisis preparation processes, involving information acquisition, collaboration, planning, and decision-making in a multi-agency setting. The main task of the participants in the gaming session was to develop a crisis preparation plan, either for their agency or for the safety region. In order to develop such a plan, each participant needed PIs (provided in the dashboards) and context information (simulated by the facilitators). Depending on the agency of the participant (fire department, ambulance services etc) and the level of crisis preparation (operational, tactical and strategic) each participant interacted with a different dashboard. The context and drivers for the crisis preparation plans were simulated based on a predefined script, instructions, and paper messages. The session simulated several potential crisis events that have occurred or may still occur in a hypothetical safety region, each requiring multi-agency crisis preparation. During the gaming simulation, the qualitative and quantitative data gathering instruments resulted in data that is discussed next.
## 4.2 Qualitative dashboard evaluation: findings of the focus group sessions

Focus groups reflected on the experience with the dashboards during the gaming simulation. The data generated was of a qualitative nature. A list of observations recorded by the facilitators stimulated the participants to share their opinions about the dashboard they used for crisis preparation. The first focus group session (after round 1) was dominated by discussions surrounding the graphical user interface (GUI) of the dashboards. While the majority of participants were positive on the GUI design, some participants pointed to the problem of information overload and complexity due to the "many performance indicators on a single screen." The second focus group session (after game round 2) was focused on the structure of the PIs in the dashboards. More specifically, the participants reflected on the alignment of the PIs to the actual process of crisis preparation. In several cases, the participants suggested that the sequence of the PIs needed to be changed in accordance to the actual process of crisis preparation for their respective agency. Moreover, some participants mentioned that the dashboard developers did not accurately understand the individual sub processes of crisis preparation. Since the alignment of PIs to the sub-processes was important, the facilitators sketched a more accurate framework of the sub processes together with the participants. Based on the resulting framework, several elements of the dashboards were restructured before the start of round 3 on day 2 of the gaming simulation. The final focus group session took place after round 3. During this focus group, the strengths, weaknesses, opportunities, and threats (SWOT) of using the developed dashboards were discussed using a SWOT analysis. While the majority of participants acknowledged the value of dashboards for multi-agency disaster preparation, there were some mixed feelings regarding the standardization and enforcement of these dashboards across all the safety regions in the Netherlands. The issue here was that none of the safety regions in the Netherlands were the same or comparable in terms of capacity for handling crises. Moreover, every safety region faces different risks and potential crisis. Hence, the thresholds for the PIs needed to include a correction for several characteristics of a safety region, including the size, risks profile, and capacity of that specific region.

## 4.3 Quantitative dashboard evaluation: findings from questionnaires

In order to capture some quantitative, user generated data from the evaluation of the proposed dashboards, we employed questionnaires. Our purpose with the survey was to collect data on the individual level of satisfaction with the dashboards, their ability to aid in crisis preparation, and the intention of the individual participants to use the dashboards in practice. We administered short (one page, single sided) paper questionnaires at the end of the gaming simulation (day 2). We employed questionnaire items from two theoretical models that explain the satisfaction and success of technology: (1) the Information Systems Success (ISS) model by Delone and Mclean [18] and (2) the Technology Acceptance Model (TAM) [19]. Both ISS and TAM contain well agreed upon and frequently tested questionnaire items for evaluating information systems. The items listed in table two were measured using a seven-point Likert scale ranging from 1= strongly disagree to 7 = strongly agree. We
analyzed the questionnaire data in order to obtain a picture of the satisfaction with and the intention to use dashboards. SPSS 17.0 yielded frequency tables, means (average values), and standard deviations for respondents' answers on the 7-point scale. The analysis provided an insight into the numbers of respondents associated with different values for a variable (criterion), the average value for each criterion—which could be considered an indication of the weight attached by the respondents to each of the different criteria used in performance evaluation—and the dispersion of the respondents' answers. Table 3 summarizes the questionnaire items and findings (n=22).

<table><caption>**Table 3.** Some questionnaire items and scores (measured using a 7-point Likert scale)</caption><thead><tr><td>Nr</td><td>Construct</td><td>Item Question</td><td>Mean</td><td>Standard deviation</td></tr></thead><tbody><tr><td>1</td><td>Collaboration1</td><td>The dashboard stimulated me to collaborate with the other domains in the safety region.</td><td>4,77</td><td>1,152</td></tr><tr><td>2</td><td>EaseofUse1</td><td>It would be easier if the PIs are clustered according to the primary processes of my organization.</td><td>5,55</td><td>1,605</td></tr><tr><td>3</td><td>EaseofUse2</td><td>The user interface (screen, buttons) was intuitive and easy to use.</td><td>5,48</td><td>1,167</td></tr><tr><td>4</td><td>TaskSupport1</td><td>The information provided via the dashboard was relevant (directly useable for executing my tasks).</td><td>4,82</td><td>1,332</td></tr><tr><td>5</td><td>TaskSupport2</td><td>Use of the dashboard leads to information overload (too much information)</td><td>3,43</td><td>1,502</td></tr><tr><td>6</td><td>TaskSupport3</td><td>The dashboard was stable and always available</td><td>4,68</td><td>1,249</td></tr><tr><td>7</td><td>Useability1</td><td>The information provided via the dashboard was easy to understand</td><td>4,86</td><td>1,283</td></tr><tr><td>8</td><td>Useability2</td><td>The refresh frequency of the dashboard was insufficient (yearly instead of monthly).</td><td>4,91</td><td>1,477</td></tr><tr><td>9</td><td>Useability3</td><td>For improved usability, the indicators on the dashboard need to be clustered in themes.</td><td>6,41</td><td>0,734</td></tr><tr><td>10</td><td>Preparation1</td><td>The information provided via the dashboard helped me to prepare for crisis response.</td><td>4,86</td><td>1,246</td></tr><tr><td>11</td><td>Preparation2</td><td>The dashboard provided me with valuable information for executing my individual tasks.</td><td>4,45</td><td>1,625</td></tr><tr><td>12</td><td>Preparation3</td><td>The dashboard provided me with valuable information for our group tasks.</td><td>5,18</td><td>1,140</td></tr><tr><td>13</td><td>Satisfaction1</td><td>I am satisfied with the dashboard.</td><td>4,86</td><td>1,153</td></tr><tr><td>14</td><td>Satisfaction2</td><td>I believe that the adoption of the dashboard would lead to improved crisis response.</td><td>5,64</td><td>.848</td></tr><tr><td>15</td><td>Intention2use1</td><td>In the future, I would like to use the dashboard in practice.</td><td>5,95</td><td>.653</td></tr><tr><td>16</td><td>Intention2use2</td><td>I will use the dashboard in the future, but only if it is used in other safety regions.</td><td>3,91</td><td>1,998</td></tr><tr><td>17</td><td>Usefulness1</td><td>The dashboard fulfilled my information needs.</td><td>3,82</td><td>1,259</td></tr><tr><td>18</td><td>Usefulness2</td><td>My information needs were beyond the information provided via the dashboard.</td><td>5,36</td><td>1,049</td></tr><tr><td>19</td><td>Usefulness3</td><td>The capability to add or remove indicators improves the usefulness of the dashboard.</td><td>6,18</td><td>.795</td></tr></tbody></table>
In total 22 out of the 27 (81%) participants remaining at the end of the second gaming-simulation day returned a completed questionnaire. From this sample, four respondents represented the fire department, four the medical services, three the emergency control room, three the central financial department of the safety region, three the Crisis Management Planning Centre, two the Safety Region management and three the Board of Directors (the Mayors). The questionnaire contained nine constructs.

The results of the questionnaire indicate that the relief agency managers found that the dashboard did improve their ability to prepare for a crisis. Items number 5 (task support), 11 (preparation) and 16 (intention to use) show the largest standard deviation in respondent scores. From the scores in the table we can conclude that the operators of the dashboards were not only satisfied with the dashboards, but also found the dashboards useful when preparing for a crisis. The majority of respondents have also indicated that they intent to use the dashboard in the future (if they were to be implemented). Based on the results of the questionnaire (high scores on usefulness, organizational impact, task-support and intention to use), we regard the dashboards developed accepted from a TAM perspective and successful from an ISS perspective.

# 5 Conclusion and discussion: experiences from the Living Lab

The main deliverable of the Aristoteles Living Lab is a set of seven dashboards for multi-agency crisis preparation. Key in the development of these dashboards was user co-creation, a process in which (future) users of the proposed dashboards were actively involved in a Living Lab. This paper contributes a description and discussion of a real-world development trajectory that, due to its explorative nature, required several research stages with professionals. The findings from both the qualitative and quantitative evaluation suggest that users were overall satisfied with the dashboards and show intention to use these in practice. Moreover, the majority of participants felt the dashboard did help them prepare for the eventuality of a crisis during the gaming session. Yet, the participants in the gaming simulation phase of the Living Lab suggest that the formulation and clustering of the performance indicators require better alignment with the context of use. We consider this alignment as one of the major challenges for further research, especially since the process of crisis response is very difficult to capture and specify in a general workflow.

Even though we collected both qualitative and quantitative data on the value of the proposed dashboards allowing us to triangulate some of our findings, the relatively small number of participants in the evaluation phase limits us in generalizing our findings. Having acknowledged this limitation, a Living Lab does allow synthesizing some experiences in the development of dashboards for disaster preparation. The experiences include the design trade-offs that need to be made by dashboard architects and are outlined in the Table 3. For scholars, these experiences may be used to formulate more specific propositions and hypotheses for future research. For practitioners, these experiences may be used as guidelines for developing dashboards. One of the main experiences is that the high level of uncertainty regarding the final set of performance indicators and the corresponding norms demands flexibility in the dashboard architecture beyond the evaluation stage.
<table><caption>**Table 3.** Experiences gained from the Living Lab</caption><thead><tr><td>Experience</td><td>Trade-off</td><td>Explanation</td></tr></thead><tbody><tr><td>Maximize stakeholder involvement</td><td>Speed of dashboard development process versus level of commitment</td><td>Involving all stakeholders in the Living Lab might reduce the speed of this process since each stakeholder has its own goals and (technical) preferences. Yet, it is crucial not to neglect the wishes of stakeholders who might lobby against the proposed dashboards.</td></tr><tr><td>Maintain open and flexible (fluid) dashboards for user co-creation</td><td>Hard coded versus flexible dashboards</td><td>In contrast to traditional system design processes, user co-creation requires flexible dashboards, PI sets, and thresholds (underlying performance norms) even during the evaluation phase.</td></tr><tr><td>Communicate problems in PI formulation</td><td>Granularity: detailed or abstract PIs?</td><td>For user co-creation, dashboard architects need to communicate problems regarding the PI formulation and evaluation process.</td></tr><tr><td>Generate “look &amp; feel” moments in the Living Lab</td><td>Dry runs or live runs?</td><td>Organize real life sessions (i.e., using focus groups and gaming simulations) to allow users to obtain a practical understanding of the implications of PI.</td></tr><tr><td>Show intention to accommodate suggestions</td><td>Closed or fixed dashboards?</td><td>User co-creation demands that the feedback and suggestions of stakeholders is implemented as soon as possible. This highlights user involvement in the dashboard development process.</td></tr><tr><td>Allow customization of dashboard representations</td><td>Options for GUI personalization by the user: fully or non-customizable interfaces?</td><td>Users should be able to choose the style of visual representation (bar charts, graphs, numeric tables) depending on the task at hand (i.e., work flow management data, payroll, human resources, material management).</td></tr><tr><td>Predefine the level of information load per PI according to roles and tasks</td><td>High or low-level information accuracy?</td><td>Too little information may lead to insufficient task support whereas too much information may lead to information overload. Consequently, dashboards should help relief agency managers by providing only the necessary information, but also attracting the attention to information easily ignored.</td></tr></tbody></table>

Based on the experiences listed in Table 3, we conclude that the development of dashboards is a difficult endeavor as it requires a constant balancing act on trade-offs. The Living Lab involves several stakeholders that co-decide about the trade-offs, and consequently construct the project specific benchmark for success. Moreover, previous research has not reported comparable dashboard development efforts that we could draw upon. We found that the Living Lab approach was useful for the development and evaluation of dashboard involving many stakeholders. User co-creation, one of the main characteristics of Living Labs, was particularly important in the dashboard design and evaluation process. User co-creation was particularly instrumental for dealing with uncertainty regarding dashboard elements, PIs, thresholds and so on. Users in the Living Lab appreciated the interaction between the various stakeholders, the use of prototypes, and the look and feel experiences generated during the gaming simulation. As such, we recommend the use of Living Labs in e-government when dealing with these types of complex problems involving many actors and uncertain (future) user needs.
References

1. Layne, K., Lee, J.: Developing fully functional E-government: A four stage model. Government Information Quarterly 18 (2001) 122-136

2. Bharosa, N., Janssen, M., Groenleer, M., van Zanten, B.: Transforming crisis management: field studies on network-centric operations. In: Wimmer, M. (ed.): 8th International Conference on E-Government, DEXA EGOV 2009. Lecture Notes in Computer Science, Lintz, Austria (2009)

3. Boin, A., 't Hart, P., Stern, E., Sundelius, B.: The Politics of Crisis Management: Public Leadership Under Pressure. Cambridge University Press (2005)

4. Carter, N., Klein, R., Day, P.: How organisations measure success: The use of performance indicators in government. Routledge, Taylor & Francis Group (1995)

5. Clarke, S.: Your Business Dashboard: Knowing When to Change the Oil. The Journal of Corporate Accounting & Finance 16 (2005) 51-54

6. Adam, F., Pomerol, J.C.: Developing Practical Decision Support Tools Using Dashboards of Information. In: Burstein, F., Holsapple, C.W. (eds.): Handbook on Decision Support Systems 2. Springer Berlin Heidelberg (2008) 151-173

7. Few, S.: Information Dashboard Design: The Effective Visual Communication of Data. O'Reilly Media, Inc (2006)

8. Dover, C.: How Dashboards Can Change Your Culture. Strategic Finance 86 (2004) 43-48

9. Gitlow, H.: Organizational Dashboards: Steering an Organization Towards its Mission. Quality Engineering 17 (2005) 345-357

10.Resnick, M.L.: Situation Awareness Applications to Executive Dashboard Design. 47th Annual Conference of the Human Factors and Ergonomics Society (2003) 449-453

11.Pallot, M.: The Living Lab Approach: A User Centred Open Innovation Ecosystem. Webergence Blog, Vol. 2010 (2009)

12.Lama, N., Origin, A.: Innovation ECOSYSTEMS: Services engineering & living labs a dream to drive innovation? : (2006)

13.Følstad, A.: Living labs for innovation and development of information and communication technology: A literature review. The Electronic Journal for Virtual Organizations and Networks 10 (2008) 99-131

14.Erikkson, M., Niitamo, V.P., Kulkki, S.: State-of-the-art in utilizing Living Labs approach to user-centric ICT innovation - a European approach. (2005)

15.Morrissey, R.: Tailoring Performance Dashboard Content. Business Intelligence Journal 12 (2007)

16.Duke, R.D., Geurts, J.: Policy games for strategic management. Dutch University Press, Amsterdam, The Netherlands (2004)

17.Meijer, S.M., Hofstede, G.J., Omta, S.W.F., Beers, G.: The organization of transactions: research with the Trust and Tracing game. Journal on Chain and Network Science 8 (2008) 1-20

18.Delone, W., McLean, E.: Information Systems Success: the quest for the dependent variable. Information Systems Research 3 (1992) pp.60-95

19.Davis, F.D.: User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. International Journal of Man-Machine Studies 38 (1993) pp.475-487