# Practitioners' Perceptions of Fitness to Task of a Leading Disaster Response Management Tool

Hans J. Scholl, Eric E. Holdeman

<figure><img src="image_1.png" /></figure>

## To cite this version:

Hans J. Scholl, Eric E. Holdeman. Practitioners' Perceptions of Fitness to Task of a Leading Disaster Response Management Tool. 20th International Conference on Electronic Government (EGOV), Sep 2021, Granada, Spain. pp.117-133, 10.1007/978-3-030-84789-0_9. hal-04175110

HAL Id: hal-04175110

https://inria.hal.science/hal-04175110

Submitted on 1 Aug 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Practitioners' Perceptions of Fitness to Task of a Leading Disaster Response Management Tool

Hans J. Scholl¹[0000-0002-6168-3445] and Eric E. Holdeman²

¹University of Washington, Seattle, WA, United States
jscholl@uw.edu

²Eric Holdeman & Associates, Puyallup, WA, United States
ericholdeman@ericholdeman.com

**Abstract.** Crisis Information Management Systems (CIMS) have been used in Emergency and Disaster Response Management for decades. However, while these systems have emerged and improved over time, they still appear to provide lower efficacy when incidents become more complex, and, in particular, when used in the context of multijurisdictional responses to large and growing incidents and extreme events. Most CIMS like E Team, Veoci, or WebEOC are commercial off-the-shelf systems (COTS), which allow for and also require from emergency response units the customization of the application to their own specific needs. This survey-informed study took a look at practitioners' experiences with one of the most widely used CIMS, that is, WebEOC. The results were mixed at best and confirm other studies, which pointed at WebEOC's lack of scalability, interoperability, network security, and ease of use. The study concludes that in the face of ever more frequent incidents of greater magnitude the case for developing and deploying securely interoperable and scalable CIMS is compelling and has to be addressed.

**Keywords:** Disaster Response Management; coordination of emergency responses; information and communication technologies; ICTs; Crisis Information Management Systems; CIMS; WebEOC; National Incident Management System; NIMS; Incident Command System; ICS; Commercial Off-the-Shelf Systems; COTS.

## 1 Introduction

Contemporary commercial Crisis Information Management Systems (CIMS), it has been suggested [1, 2], support emergency managers and responders relatively well when used in response to smaller, every-day, and geographically isolated incidents. When, however, multiple jurisdictions and different levels of government agencies need to coordinate their responses, they appear to exhibit limitations and rigidities in terms of interoperability, scalability, reliability, network security, and ease of use. Unfortunately, not only when different vendors' systems have to interact, but rather also when the same system such as WebEOC has to scale up to meet the needs of a more demanding multi-level and multi-jurisdictional context [3], this experience of constraints and lack of scalability seems to be commonplace [4, 5].
As discussed elsewhere, given the wide range of incidents in terms of scale, scope, and duration [6] spanning from local emergencies such as leaks of hazardous materials or a building afire to large-scale, large-scope, and long-duration catastrophes such as the Indian Ocean Earthquake and Tsunami of 2004, the Galveston Hurricane of 1900, or the East Japan Earthquake and Tsunami of 2011, the coordination, integration, and scalability of operations is essential to the effectiveness of the response [7, 8]. In the United States, the National Response Framework (NRF) and the National Incident Management System (NIMS) with its Incident Command Structure (ICS) were designed exactly for the purpose of providing a common terminology and a framework of principles, practices, processes, which would be scalable, flexible, and comprehensive enough to guide responders of any discipline along the whole spectrum of possible disasters (ibid.). While these response frameworks entail a certain degree of organizational standardization, the information management portion of the framework and its supporting information and communication technologies (ICTs), that is, the CIMS would have to carry the burden, as the NIMS/ICS planners anticipated,

> "Communications and information systems should be designed to be flexible, reliable, and scalable in order to function in any type of incident, regardless of cause, size, location, or complexity. They should be suitable for operations within a single jurisdiction or agency, a single jurisdiction with multiagency involvement, or multiple jurisdictions with multiagency involvement. Communications systems should be applicable and acceptable to users, readily adaptable to new technology, and reliable in the context of any incident to which emergency management/response personnel would be expected to respond" [7, p. 24].

By the time, these guidelines were formulated, CIMS such as WebEOC were already in use at all levels of government across the United States, and although WebEOC was still far from having become a kind of de-facto standard, a rather wide variety of non-interoperable CIMS was already in use around the Country. This, however, began to slowly change when in 2012 the US Federal Emergency Management Agency (FEMA) also adopted WebEOC for their internal use [9]. Yet, only a few years into FEMA's use of this particular CIMS, it became evident that for security and performance reasons the WebEOC implementation at FEMA would not interoperate with State, Territory, or Tribal WebEOC sites forcing both FEMA and the respective agencies into tedious, error-prone, and time-consuming double work for necessary data exchanges [10]. It immediately follows that during responses to larger incidents, when the coordination and collaboration between and among agencies, both vertically and horizontally, are badly needed such bottlenecks would be counterproductive and costly. This study's intent was to find out, document, and analyze how practitioners experience their work with WebEOC during a response with the aim of better understanding the efficacy and usefulness of this commercial CIMS along the entire spectrum of emergencies and disasters.

The publication is organized as follows: First, related work on WebEOC in the academic literature is reviewed. Next, research questions and the methodology are
detailed. Then, the findings for each research question are presented followed by a
discussion of the findings. Finally, concluding remarks and directions for future
research are presented.

## 2 Related Work

While professional response organizations in the United States have chosen from and implemented a wide range of COTS CIMS for the purpose of supporting their respective response operations, WebEOC appears to have become the most proliferated CIMS [4, 11]. Interestingly, in academic research, although CIMS are at the core of modern information management in emergency response, they have not yet systematically been assessed and evaluated in terms of their efficacy over the entire spectrum of incident responses. As a recent study suggested, “the role of information management tools used in <emergency management>... needs further investigation,” which according to the authors required to include “studying differences between centralized control and distributed participation; incorporating multiple incident data into a visually informative form for decision makers (e.g., hazardous conditions); and improving designs suitable for updating information in a timely manner” [12, p. 10]. Nevertheless, some research, even with regard to WebEOC, has been conducted while the overall picture has remained spotty. The first well-known quasi-academic comparison of contemporary CIMS was conducted as early as 2002 by the Hart study, which was sponsored by the Department of Justice's National Institute of Justice (NIJ). The study performed a 106 “features” comparison of then contemporary CIMS, among which WebEOC scored highest with 102 desirable “features” identified [13].

A number of WebEOC-related studies went down a similar pathway, when investigating features (such as “boards”) and their relative usefulness in incident responses [11, 14-17]. Some studies took a high-level descriptive approach without concerning themselves with any technology or feature details [16, 18]. WebEOC was also found as a unifying connecting link in private-public partnerships between business communities and government agencies [2, 19].

While studies like the former portrayed WebEOC's versatility, others emphasized more critical findings. Around the time that the NIMS/ICS designers detailed their recommendations regarding necessary interoperability, interconnectivity, and scalability requirements for CIMS (as quoted in the introduction), academic research also came to similar conclusions [5, 20]. Along these lines, WebEOC was found incomplete in terms of the availability of needed information [21] also with regard to information sharing between and among different agencies [22]. During a multi-jurisdictional response to a major landslide disaster, WebEOC implementations at Federal level (FEMA) and State levels were found unable to interoperate in most basic ways for security concerns on either side [23]. Similarly, in the simulation of a catastrophic incident, that is, under artificial exercise conditions when no damage of critical infrastructure had actually occurred or was even assumed under the simulated scenario, the interoperation and information sharing between a State-operated WebEOC site and two dozen county WebEOC sites broke completely down under the sheer load of requests [24].
Besides these serious load-related issues, other reports found a lack of automatic information summarization technologies implemented in WebEOC [15], which made it hard for responders to see the forest for the trees, once an incident response began growing, so that information had to be manually aggregated on paper to be useful [25]. The latter study also found “poor interoperability” within the network infrastructure [25, p. 9] resulting in poor information sharing exacerbated by ineffective ex-ante staff training and unaddressed communication gaps.

Other studies found that in order to make good use of WebEOC, the incident response had to accommodate to the “tool” rather than that the tool accommodated to the demands and needs of the response [26]. The same study also reported that WebEOC’s customizability, while giving the respective agency the flexibility to tailor the tool to its specific needs, by so doing it also sacrifices standardization and compatibility with other WebEOC implementations. Many agencies, particularly resource-poor ones, may even have great difficulty with setting up WebEOC in a tailored fashion [5, 26].

When analyzing the efficacy of a particular CIMS in response to an incident, scale, scope, and duration of this particular incident present the first benchmarks to consider. As mentioned in the introduction, a CIMS, which performs reasonably well in the response to incidents of small scale, small scope, and short duration may not perform as well when scale, scope, and duration of the incident increase, let alone when catastrophic proportions are reached. However, few studies such as Son et al. on CIMS efficacy have taken this consideration into account. When, for example, taking Fischer’s scale [6], which categorizes emergencies and disasters in a range from “DC (for disaster category) 1” to “DC 10” with “DC 1” standing for an everyday local and small incident and “DC 10” representing a catastrophe of extraordinary magnitude and annihilation, it appears intuitively evident that a CIMS has to be extremely scalable to cover this enormous range.

In summary, the efficacy of existing CIMS, and especially, WebEOC, in emergency and disaster response management is understudied; therefore, it has remained an open question of whether or not current CIMS, and here, WebEOC, are fit to task, in particular, when it comes to larger and dynamically growing incidents towards the upper end of the Fischer scale.

# 3 Research Questions and Methodology

## 3.1 Research Questions

As the literature review illustrated the gap of understanding with regard to the efficacy of CIMS, and specifically WebEOC, in practical disaster response management is wide. Moreover, it will be a potentially life-saving and likely disaster-mitigating contribution if this known gap in understanding could be narrowed. It also appears that the most important stakeholders, that is, disaster responders would bring first-hand practical experience to the table, when it comes to the efficacy of WebEOC, which leads to the following two research questions:
**Research Question #1 (RQ #1):**

How do professional emergency responders perceive the efficacy and fitness to task of WebEOC in emergency response management?

**Research Question #2 (RQ #2):**

What specific, if any, concerns regarding WebEOC (and its use) do professional emergency responders express in emergency response management?

## 3.2 Data Selection and Analysis

*Instrument.* Owed to the paucity of research on the subject and the concurrent absence of a guiding theoretical framework, the inquiry had to be of exploratory nature. For this purpose, an online (Google Forms) fourteen-question survey was devised. All questions except the last were either single or multiple choice questions. The first two questions established demographics (affiliation, WebEOC licensing status). The next eleven questions queried about use and performance aspects of WebEOC. The last question was free-format and open-ended, which gave participants an additional opportunity to enter their own observations in a narrative.

*Sample.* The intention of the researchers was to reach out and receive feedback from both the managerial and professional levels of disaster responders around the country. As a reminder, this inquiry did not seek to statistically test any hypotheses regarding the usefulness of WebEOC in disaster response. It rather intended to explore the perception of usefulness of WebEOC on part of professional disaster responders who had gained practical experience with this particular CIMS, which suggested a convenience sample was to be used [27]. As a long-term practitioner and a leader in the field, the second author has run a regular and well-respected blog for several years, which is read by a large audience of response professionals. In order to attract responses, the survey was attached to an opinion editorial that the second author posted on his blog site [28]. The tone of the blog was critical towards WebEOC, which was hoped to prompt professional responders into reacting and taking the survey, either for reasons of strong disagreement, or for the opposite.

*Data Collection.* The vast majority of the data were received and collected within two and a half weeks after the publication of the opinion piece. Very few responses were entered weeks after the publication. A total of 83 responses were received, and 48 respondents also took the time to enter a narrative, some of which was extensive and rich. Half of the respondents were County-level responders, 26.3% State-level responders, 13.1% Municipal Government-level responders, just under 5% Federal-level responders, 2.4% other governmental institution-level responders, and 3.6% were non-governmental-organization-based responders. The vast majority of responders (72.6%) represented organizations that had active WebEOC licenses, and 17.9% of responders represented organizations that previously held a WebEOC license.

*Data Analysis and Coding.* Data were analyzed in an open coding approach [29, 30], in which tentative concept labels were attributed to chunks of texts and their particular attributes. The concept labels were connected with regard to their relationships to each other in an axial-coding exercise [31] and compared to the
results of the eleven single and multiple-choice survey questions for plausibility, consistency, and further explanation.

*Neutrality/Impartiality*. This research has not been funded nor otherwise supported by any commercial or other vested interest.

# 4 Findings

In the following the findings are presented in the order of the research questions. The findings from both the single/multiple-choice questions and the free-format narratives are integrated in each findings subsection.

**Ad Research Question 1 (RQ #1)** — (How do professional emergency responders perceive the efficacy and fitness to task of WebEOC in emergency response management?)

The first survey question attempted to establish the overall satisfaction or dissatisfaction of survey respondents with WebEOC. Without undue speculation it was assumed that responders' satisfaction or dissatisfaction with WebEOC would largely correspond to the system's perceived efficacy and fitness to task. It was found that 33.7% of respondents were either very satisfied or at least somewhat satisfied whereas a majority of respondents (53%) across all groups were either very dissatisfied or somewhat dissatisfied, while 18.1 percent of respondents fell into neither camp. In other words, with only about one third of respondents expressing satisfaction of some kind with the system, WebEOC's efficacy and fitness to task in emergency and disaster response management appears to be called into question for the most part. However, based on the demographic and other data derived from the survey, it was possible to provide sharper contours and more granular detail for painting this mostly unfavorable overall picture of WebEOC performance in US emergency and disaster response management.

As Figure 1 shows when breaking down the distribution of satisfaction and dissatisfaction along government levels, WebEOC-related satisfaction is strongest at Federal and State levels, whereas on County and municipal levels dissatisfaction prevails. It is noteworthy, though, that while the survey produced only four responses from the Federal level, the four responses were widely spread with regard to the degree of satisfaction, and no Federal-level response indicated the highest level of satisfaction with WebEOC.
<figure><img src="image_4.png" /><figcaption>Figure 1. WebEOC User Satisfaction/Dissatisfaction per Government Level</figcaption></figure>

<table><caption>Table 1. WebEOC User Satisfaction/Dissatisfaction Relative to Other Factors</caption><thead><tr><th>Variable Values<br>(Variables: Frequency, Ease of Use, Functionality, & Customization)</th><th>Very or somewhat<br>dissatisfied (%)</th><th>Very or<br>somewhat<br>satisfied (%)</th></tr></thead><tbody><tr><td>Daily Use</td><td>17.9%</td><td>60.7%</td></tr><tr><td>Weekly Use</td><td>28.2%</td><td>14.3%</td></tr><tr><td>Monthly Use</td><td>15.4%</td><td>14.3%</td></tr><tr><td>Infrequent Use</td><td>38.5%</td><td>10.7%</td></tr><tr><td>Very complicated</td><td>37.5%</td><td>0.0%</td></tr><tr><td>Somewhat complicated</td><td>32.5%</td><td>37.0%</td></tr><tr><td>Neither complicated nor easy</td><td>20.0%</td><td>22.2%</td></tr><tr><td>Somewhat easy</td><td>10.0%</td><td>29.6%</td></tr><tr><td>Very easy</td><td>0.0%</td><td>11.1%</td></tr><tr><td>Basic Functions</td><td>26.3%</td><td>17.9%</td></tr><tr><td>Advanced Functions</td><td>73.7%</td><td>82.1%</td></tr><tr><td>No/Little Customization</td><td>28.2%</td><td>11.1%</td></tr><tr><td>Some Customization</td><td>25.6%</td><td>25.9%</td></tr><tr><td>Substantial Customization</td><td>25.6%</td><td>33.3%</td></tr><tr><td>Extensive Customization + Add-ons</td><td>20.5%</td><td>29.6%</td></tr></tbody></table>

Taking County and municipal levels together, overall dissatisfaction with WebEOC was more than twice as frequent as was overall satisfaction with the tool. Among the variables, which might influence the degree of satisfaction or dissatisfaction, the authors suspected “frequency of use,” “ease of use,” “functionality,” and “degree of customization” (see Figure 1). Upon inspecting the percent values, one might tend at first sight to associate, for example, far more frequent, that is, “daily use” with higher degrees of satisfaction; as Figure 1 shows, only 17.9% of “somewhat or very dissatisfied” respondents indicated that they used WebEOC on a daily basis, while,
in contrast, outright 60.7% of “somewhat or very satisfied” respondents suggested they used the system on a daily basis. One might argue that more frequent use leads to, or, at least, illustrates higher degrees of satisfaction. However, regression analyses on all independent variables specified above and their combinations did not produce any statistically significant prediction of the dependent variable (satisfaction/dissatisfaction).

Despite this particular finding, it is noteworthy that WebEOC customization is the greater the higher the level of government (see Table 1). For the Federal level, substantial WebEOC customization was reported in all cases, on State level, customization is over 82% with half of this attributed to “extensive customization with add-ons.” On County and municipal levels, customization of WebEOC is also found in or slightly above 70% the responses; however, on the municipal level no “extensive customization with add-ons” is reported.

<table><caption>**Table 2.** WebEOC Customization per Government Level (%)</caption><thead><tr><th>Degree of Customization</th><th>Federal</th><th>State</th><th>County</th><th>Municipal</th></tr></thead><tbody><tr><td>Little or no customization</td><td>0.0%</td><td>13.6%</td><td>25.6%</td><td>30.0%</td></tr><tr><td>Some customization</td><td>0.0%</td><td>18.2%</td><td>23.1%</td><td>50.0%</td></tr><tr><td>Substantial customization</td><td>100.0%</td><td>22.7%</td><td>30.8%</td><td>20.0%</td></tr><tr><td>Extensive customization/add-ons</td><td>0.0%</td><td>40.9%</td><td>20.5%</td><td>0.0%</td></tr><tr><td>Not sure</td><td>0.0%</td><td>4.5%</td><td>0.0%</td><td>0.0%</td></tr><tr><td>Totals</td><td>100.0%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr></tbody></table>

<table><caption>**Table 3.** WebEOC Functionality Usage per Government Level (%)</caption><thead><tr><th>Functionality</th><th>Federal</th><th>State</th><th>County</th><th>Municipal</th></tr></thead><tbody><tr><td>1=Basic Functions</td><td>0.0%</td><td>0.0%</td><td>30.8%</td><td>70.0%</td></tr><tr><td>2=Advanced Functions</td><td>100.0%</td><td>100.0%</td><td>66.7%</td><td>30.0%</td></tr><tr><td>3=Not sure</td><td>0.0%</td><td>0.0%</td><td>2.6%</td><td>0.0%</td></tr><tr><td>Totals</td><td>100.0%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr></tbody></table>

With regard to functionality, again the higher the government level the more the advanced functionality of WebEOC is employed. At municipal level, overwhelmingly basic functionality of WebEOC is adopted, which illustrates an enormous gap of sophistication and experience between the local level and even the next higher level (County), let alone the State and Federal levels (see Table 2).

One of the most important, if not the most important task in emergency management is gaining and maintaining *Situational Awareness (SA)*, which is the prerequisite for developing and also maintaining a *Common Operating Picture (COP)*. Fully developed and vetted SA/COP are at the core of any effective and successfully targeted response to any incident [7, 32]. In this particular context, the
detailed geographic location of incident-related information is essential. Many response units employ highly specialized Geographic Information Systems (GIS) such as the Environmental System Research Institute's ArcGIS. However, WebEOC also comprises a mapping component of its own and affords some GIS record integration with ArcGIS, although the WebEOC mapping component lacks the sophistication of ArcGIS.

<table><caption>**Table 4.** Usage of WebEOC Mapping per Government Level (%)</caption><thead><tr><th>Usage of WebEOC Mapping</th><th>Federal</th><th>State</th><th>County</th><th>Municipal</th></tr></thead><tbody><tr><td>Mapping never used</td><td>75.0%</td><td>22.7%</td><td>48.7%</td><td>60.0%</td></tr><tr><td>Mapping rarely used</td><td>0.0%</td><td>27.3%</td><td>20.5%</td><td>40.0%</td></tr><tr><td>Not sure</td><td>0.0%</td><td>9.1%</td><td>2.6%</td><td>0.0%</td></tr><tr><td>Mapping regularly used</td><td>25.0%</td><td>22.7%</td><td>17.9%</td><td>0.0%</td></tr><tr><td>Mapping key functionality</td><td>0.0%</td><td>18.2%</td><td>10.3%</td><td>0.0%</td></tr><tr><td>Totals</td><td>100.0%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr></tbody></table>

As Table 3 demonstrates, WebEOC-based mapping is rarely if ever used at municipal level, and also at Federal level its usage is relatively low, which in this latter case is most likely attributable to the use of more powerful GIS at Federal level. But on County and State levels, this particular mapping functionality is never or rarely used in almost 70%, or almost 50%, of the cases, respectively. Similar to the Federal level, it is likely that both County and State responders utilize more powerful and more specialized tools like ArcGIS instead. As before, a major gap in functionality utilization, and consequently, sophistication with regard to WebEOC-based generation and preservation of SA/COP appears to exist between municipal levels of government and higher levels.

<table><caption>**Table 5.** Intra-/extra-jurisdictional Connectivity per Government Level (%)</caption><thead><tr><th>Connectivity</th><th>Federal</th><th>State</th><th>County</th><th>Municipal</th></tr></thead><tbody><tr><td>Not connected (intra-jurisdictional)</td><td>75.0%</td><td>15.0%</td><td>23.7%</td><td>22.2%</td></tr><tr><td>Some connected (intra-jurisdictional)</td><td>25.0%</td><td>65.0%</td><td>57.9%</td><td>66.7%</td></tr><tr><td>All connected (intra-jurisdictional)</td><td>0.0%</td><td>20.0%</td><td>18.4%</td><td>11.1%</td></tr><tr><td>Totals (intra-jurisdictional connectivity)</td><td>100.0%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr><tr><td>Not connected (extra-jurisdictional)</td><td>75.0%</td><td>10.5%</td><td>30.8%</td><td>0.0%</td></tr><tr><td>Some connected (extra-jurisdictional)</td><td>25.0%</td><td>36.8%</td><td>43.6%</td><td>88.9%</td></tr><tr><td>All connected (extra-jurisdictional)</td><td>0.0%</td><td>52.6%</td><td>25.6%</td><td>11.1%</td></tr><tr><td>Totals (extra-jurisdictional connectivity)</td><td>100.0%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr><tr><td>Upstream log in</td><td>n/a</td><td>40.9%</td><td>27.8%</td><td>33.3%</td></tr><tr><td>No upstream log in</td><td>n/a</td><td>59.1%</td><td>72.2%</td><td>66.7%</td></tr><tr><td>Totals (upstream log-in connectivity)</td><td>n/a</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr></tbody></table>

Problems with connectivity have been a known WebEOC characteristic, which encompass network load issues, connection security issues, slow or no responses, connection failures, among others [22-24, 26, 33]. The survey instrument distinguished between same-jurisdiction connections (including resource requesting
and tracking) and cross-jurisdictional connections (also, including resource
requesting and tracking). Since most respondents were believed to be non-experts
on ICT network-related matters a control question was included that prompted for
the type of establishing connections to WebEOC systems in other jurisdictions
(upstream log-in for access). For example, FEMA does not allow for their WebEOC
implementation to directly interoperate with States' WebEOC implementations.
Rather State responders have to remotely log on to the FEMA system via a secure
connection, where an account for them is maintained. Many States act likewise with
their lower-ranking jurisdictions. If upstream log-in is provided as a connectivity
mechanism, then in all likelihood there is no other connectivity mechanism
established in that particular direction. It is obvious that this type of interoperation
is anything but seamless and has to be seen as an inelegant workaround. As Table 4
shows, Federal-level respondents confirmed that most of their WebEOC
implementations do not connect to other systems. In contrast, a majority of State,
County, and municipal respondents corroborate that most of their WebEOC
implementations interoperate in some fashion with other WebEOC systems, both
intra-jurisdictionally and extra-jurisdictionally. The highest numbers of upstream
log-in access were found at State level (40.9%) followed by municipalities (33.3%)
and Counties (27.8%). However, that means that in most cases no upstream log-in
appears to exist or to be used.

When looking at the narratives in survey responses, only a single highly positive
comment stood out from a State-level responder who reported on a decade-long
experience also praising the cost-benefit ratio of WebEOC. Unfortunately, the
responder did not give more details, for example, regarding the use of the system
across emergencies of different magnitudes, or regarding the coordination with other
agencies. Another State responder stated that all lower-level jurisdictions log on to
their State WebEOC system during an incident response, so that the incident is dealt
with from a unified SA/COP perspective. Another State responder referring to an
identical log-in-from-remote setup between the State and lower-level jurisdictions,
however, remarked that the system was “clunky” and not easy to use, and unlike the
other two respondents who were very satisfied with WebEOC, this respondent was
somewhat dissatisfied. Some County respondents noted that their upstream log-in
setup was a one-way street only for sharing State information downstream.
Interestingly, WebEOC’s mapping functionality was mentioned in the narrative
responses only twice: Despite one County respondent’s overall dissatisfaction with
WebEOC, this individual was highly appreciative of the integration of ArcGIS and
WebEOC, which allowed for importing some data from WebEOC into ArcGIS
(sic!). Another respondent who was neither satisfied nor dissatisfied with WebEOC
had not come across this integration functionality and rather urged for GIS
integration. Although mildly satisfied with WebEOC, but implicitly acknowledging
the system’s problems, one Federal responder stated,

> “Big Tech could solve this easily; look at the interoperability within Apple or Google apps - maps, sharing, email, messaging, browsing are instinctively linked. Imagine what they could do if given the task to package existing apps into an EM layer” (quote #1—from survey responses).
In summary, dissatisfaction with WebEOC among survey respondents was found much stronger and more outspoken than satisfaction thereof, and, in particular, the "very dissatisfied" outnumbered the "very satisfied" by a margin of more than two to one in the responses. When analyzing, which (independent) variables might have influenced these particular outcomes, neither "frequency of use," nor the "functionality used" (basic or advanced), nor the "degree of customization", nor "ease of use" were found predictors for satisfaction or dissatisfaction on part of the respondents. Furthermore, WebEOC mapping was relatively sparsely used, and certainly not as a core function, but rather as an add-in, which was connected to a more powerful GIS. Finally, overall WebEOC connectivity was only moderately implemented with States most highly engaged. Given the relatively high levels of dissatisfaction in the perception of responding practitioners WebEOC's fitness to task appears to be debatable, and it appears that the reasons for this dissatisfaction are multifold.

**Ad Research Question 2 (RQ #2)** — (What specific, if any, concerns regarding WebEOC (and its use) do professional emergency responders express in emergency response management?)

Respondents expressed most of their concerns in the open-ended narrative at the end of the survey. However, one multiple-choice survey question was geared at eliciting potential concerns regarding moving away from WebEOC. Nine choices were given including one "other" as shown in Table 6 below. The respondents marked a total of 222 choices, or on average 2.7 per respondent. It appears that no single concern stood out above all others with the exception of "budgetary concerns" with 33.3%, which was most pronounced on Federal level than on any other government level followed equally with 16.7% each by concerns regarding "going outside the norm" (sic!), "difficulty of adopting a new system," and both the institution and its internal stakeholders too "deeply invested" into the status quo with WebEOC.

<table><caption>**Table 6.** Concerns and Obstacles Perceived (When Switching from WebEOC -- in %)</caption><thead><tr><th>Concerns/Obstacles</th><th>Federal</th><th>State</th><th>County</th><th>Municipal</th></tr></thead><tbody><tr><td>State standard</td><td>0.0%</td><td>20.6%</td><td>18.8%</td><td>18.2%</td></tr><tr><td>No known alternative</td><td>0.0%</td><td>16.2%</td><td>15.8%</td><td>21.2%</td></tr><tr><td>Budgetary concerns</td><td>33.3%</td><td>11.8%</td><td>10.9%</td><td>6.1%</td></tr><tr><td>New system might not be working</td><td>0.0%</td><td>4.4%</td><td>6.9%</td><td>9.1%</td></tr><tr><td>Changing means going outside the norm</td><td>16.7%</td><td>7.4%</td><td>9.9%</td><td>18.2%</td></tr><tr><td>Difficult adoption of a new system</td><td>16.7%</td><td>16.2%</td><td>6.9%</td><td>12.1%</td></tr><tr><td>Institution deeply invested</td><td>16.7%</td><td>13.2%</td><td>10.9%</td><td>9.1%</td></tr><tr><td>Internal stakeholders deeply invested</td><td>16.7%</td><td>8.8%</td><td>9.9%</td><td>6.1%</td></tr><tr><td>Other</td><td>0.0%</td><td>1.5%</td><td>9.9%</td><td>0.0%</td></tr><tr><td><strong>Totals</strong></td><td><strong>100.0%</strong></td><td><strong>100.0%</strong></td><td><strong>100.0%</strong></td><td><strong>100.0%</strong></td></tr></tbody></table>

In lower-ranking jurisdictions the implicit or explicit de-facto standardization on WebEOC as the emergency and disaster management system represented the most
highly cited concern with 20.6% at State, 18.8% at County, and again 18.8% at municipal levels. Furthermore, at municipal level a higher-ranking concern with 21.6% was the absence of a "known alternative" to WebEOC. Since only very few "other" concerns were specified, the first eight choices in this multiple-choice question on the subject must have covered the prevailing concerns and perceived obstacles fairly comprehensively.

On the municipal level, respondents' concerns revolved around the perceived high cost of changes in system versions, system administrators, and when adding functionality by coding and recoding. Some municipal responders felt that WebEOC was oversized for their respective needs, while others bemoaned the absence of mobile versions. Quite a number of municipal responders decried the perceived lack of ease of use and straightforward task-relevant functionality. Said one respondent,

"Having used both <another system> and WebEOC, I have found WebEOC is clunkier and less user friendly. <The other system> is easier to build what is needed by the user." (quote #2—from survey responses).

And another municipal respondent explained,

"WebEOC is designed for large agencies not the majority of EM offices with one or two staff" (quote #3—from survey responses).

County respondents also criticized that WebEOC's lack of intuitiveness and ease of use, which they felt did not take into account the relatively modest ICT savviness of average emergency responders, in particular, in the more typical case that responders would not use the system frequently enough to maintain familiarity. Like respondents on municipal level, the county respondents also expressed dissatisfaction with WebEOC's lack of functionality and unintuitive user interface. In this context some respondents used explicit language to illustrate their personal frustration with WebEOC ("It sucks." "Terrible old system that has become a failure." "Outdated and obtuse system that wastes an agency's limited resources." "Lots of money spent. Still doesn't work." "After using WebEOC for numerous large and small emergencies and disasters over the past years, I give WebEOC functionality and "usability" a grade of C/C-."). Respondents on this government level also pointed at compatibility and interoperability problems, which emanated in part from a lack of standards set and followed in WebEOC implementations resulting in a wide variety of organizationally incompatible implementations. Some of these incompatibilities could be attributed to different needs at different levels of government. As one County respondent shared,

"Local EOCs who have adopted WebEOC as their software of choice in this State are buying and operating their WebEOC systems independently (State and local WebEOC systems do not interface with each other). If the locals buy their own WebEOC system, they tailor boards also to their respective EOC needs and unfortunately all the boards do no match each other at all the respective EOC levels" (quote #4—from survey responses).

On State level, the concerns of lower-ranking jurisdictions were echoed regarding the lack of "ease of use" and "functionality" as well as the high cost of maintenance including extensive training needs for coping with WebEOC's complexity.
Frustration with WebEOC was also expressed on this level in no uncertain terms (“WebEOC is very clunky.” “<WebEOC> has outlived its usefulness and should be trashed.”). As mentioned while for States “interconnectivity” exists via secure upstream single-user log-in onto FEMA’s WebEOC implementation (and not, as manifest, via a bidirectional State-WebEOC-to-FEMA-WebEOC interoperation protocol), this type of interconnection is limited in capacity and bi-directionality, which creates unpleasant bottlenecks and redundancies, in particular, in the resource request process, which employs so-called resource request forms (RRFs). As a State respondent explains,

> "Throughout our response in various disasters, missions could easily get lost as hundreds of mission requests flowed into WebEOC in a short period of time. This was problematic and challenging. We created alerts if missions were time sensitive or not updated in a timely fashion. Having no interface between our State and FEMA's WebEOC platform is problematic and time consuming as we submit RRF's, we have to submit paper RRFs to FEMA which defeats the purpose of WebEOC" (quote #5—from survey responses).

Another respondent summed up the experience with WebEOC this way,

> "The biggest problem with WebEOC is that emergency managers all too often have to manage WebEOC instead of using it as a tool (quote #6—from survey responses).

On Federal, respondents did not leave comments except for one (see quote #1 above). However, this respondent also highlighted the high licensing cost and the limitations (“single channel”) of WebEOC.

In summary, respondents from all levels of government were critical with the relatively high cost of licenses for using and maintaining WebEOC. In particular, a dearth of functionality, a deficit in true interoperability, and a lack of ease of use were criticized on all levels of government. The terms “clunky” and “outdated” were used repeatedly.

# 5 Discussion, Future Research, and Concluding Remarks

## 5.1 General Observations

Crisis Information Management Systems (CIMS) differ from other and, particularly, non-mission critical information management systems in at least two ways: (1) With regard to their specific purpose, and even more so, with respect to their centrality in the overall critical information infrastructure, CIMS must be resilient, that is, robust, resourceful, redundant, and rapid (in operational usability and response time) [7, 34]. The findings of this study along with those of earlier academic reports, however, suggest that WebEOC would not qualify as a resilient CIMS along these lines, at least not in terms of robustness nor rapidity. It rather appears to slow down in response time and even to break down under only moderate loads, which apart from security and network safety concerns also explains the lack of true multiway interconnectivity (with FEMA presenting the most prominent example).
Interconnectivity is at the core of any system's scalability. If interconnectivity is limited, then scalability is limited. With limits in scalability the respective CIMS can only be reasonably and safely used in responses to relatively small-scale incidents. (2) CIMS are supposed to be extremely easy to understand and use [7, 35], since under the typically increasingly stressful circumstances of an incident response, professional responders have no time nor do they have the stomach for struggling with idiosyncrasies and peculiarities of any given CIMS. The respective CIMS has to support the response seamlessly and without putting additional burdens on its users. However, WebEOC reportedly appears not to be in this category of seamless CIMS. Paraphrasing one respondent's words, when using WebEOC in more complex incident responses, rather frequently the tail appears to be wagging the dog.

In a nutshell, in terms of flexibility, reliability, scalability, and ease of use, WebEOC does not appear to meet the CIMS standard requirements formulated in the basic NIMS document of 2008 [7, p. 24] according to this study's findings.

## 5.2 The Need for a Widely Accepted, Resilient, and Scalable CIMS

In a recent study on the subject, Son and colleagues performed a meta-analysis of the literature [12] and confirmed earlier insights regarding resiliency in emergency management that highlighted factors including “collective sensemaking,” “team decision making,” and “interaction and coordination” (p. 10), all of which heavily rely on the availability and proper functioning of a capable and robust CIMS as a prerequisite. As has been shown elsewhere, it does not suffice that a system can actually perform certain operations under certain circumstances. It rather also matters what the respective system’s overall performance expectancy is, which is colored by past and present experiences from a human agent’s (and here professional responder’s) perspective [5]. If human agents who have to perform together as a coordinated group grow increasingly frustrated with a system’s expected performance, then the acceptance of using such a system plummets, which seems to be the case with WebEOC among numerous responders on all level of government. In other words, if WebEOC’s reputation among a large group of responders is turning unfavorable through lived and repeated negative experience, then the impact on the adoption and use of the system in the larger community of responders becomes problematic through the social influence of the disenchanted group. Conversely, if a CIMS, satisfies the performance expectations of a group of human agents, then the system’s adoption and use is strongly and more widely supported also through the group’s social influence. As seen in the findings, some responders suggested alternative COTS CIMS, while still others preferred a national initiative and a system standardized and centrally supported for all levels of government at affordable cost. The idea of using cloud-based services and existing standard tools for such undertaking might also be a viable path, which deserves study and evaluation.

It has been argued elsewhere that scalability is not only an upward affair, but rather also includes downward capabilities in case that power and networking capabilities are completely lost for an extended period of time and low-tech solutions have to be employed temporarily. While these incident scenarios might still be rare,
incidents of larger magnitude will undoubtedly encompass situations, in which, on
the one hand, multijurisdictional collaboration and coordination of the response is
badly needed, while, on the other hand, major portions of the critical (information)
infrastructure are destroyed or degraded to an extent, which makes such coordination
and collaboration extremely difficult. CIMS redundancy then means that critical
functions in such scenarios need to be performable elsewhere, and logistical support
for equipping responders on the ground who have limited or no direct connectivity
with updated stand-alone CIMS from remote sites via appropriate means of physical
transportation.

5.3 Limitations of the Study

WebEOC is a CIMS predominantly used in the United States. The results reported
here may be different for other CIMS in other countries. Also, software systems
undergo relatively frequent revisions. This study did not discriminate between
respondents reporting on the most recent version as opposed to older versions of this
particular COTS. As a result, experiences with newer versions of WebEOC might
have produced more favorable results. However, the convenience sample still reports
on "what is currently out there;" yet, a large random or systematic sample might
have produced more accurate results. While the number of respondents (83) who
fully took the survey cannot be called small, it nevertheless was not large enough to
produce highly robust results, which would lend themselves subject to elaborate
statistical analyses. The study attracted participation by attaching the Web survey to
a subject matter blog widely read by practitioners in emergency management. This
particular blog entry on WebEOC was highly critical of the system, which might
have primed and incentivized more respondents with negative than positive views
to also share their own mainly negative views on WebEOC. Therefore, besides the
sampling situation also from this latter perspective of potential bias, no claim of
results-based generalizability can be made. Furthermore, when testing for predictors
for the dependent variable (satisfaction/dissatisfaction) via regression analysis, none
of the independent variables was found as outcome predictors. This might change in
case of a larger and systematic sample. While these limitations are acknowledged,
the study nevertheless was able to document in detail considerable dissatisfaction
with WebEOC on part of practitioners on all levels of government who gave ample
comments in support of their views. The study, hence, represents a broader
exploratory step than previous studies also geared at better understanding the current
problem space involving WebEOC.

5.4 Concluding Remarks and Future Research

It has been the object of this exploration to identify and document practitioners and
system users' perceptions of fitness to task of WebEOC, the leading crisis
information management system (CIMS) in the United States. This study contributes
to the understanding of challenges and potential pitfalls when using commercial-off-
the-shelf systems (COTS) in the context of emergency and disaster response
management. While WebEOC appears to have some support among practitioners
(mainly on Federal and State levels), a far larger number of practitioners on all levels
of government was found to be highly critical of the system with respect to its
perceived high cost, difficult maintenance, low performance, insufficient
functionality, limited interoperability, and weak scalability. Since CIMS are the
backbone of effective all-hazard and all-magnitude incident responses, these
findings, which are supported by other studies, have to prompt further research, since
they suggest a serious vulnerability in the Nation's capacity to effectively cope with
emergencies and disasters, which can have adverse consequences for lives and
assets.

Future research therefore needs to focus on how CIMS can be devised that meet
the long established criteria and performance benchmarks [7, 36] and what obstacles
must be cleared in order to implement them.

References

1. P. Hanson, and C. McDougall, "Enabling Collaborative and Resilient Emergency Management Efforts: DFES and Western Australia's Adoption of a Common Operating Picture.." pp. 75-82.

2. J. Levy, and R. Prizzia, "Customizing Web-EOC Crisis Management Software to Facilitate Collaboration and Increase Situational Awareness: Advances in Business Resource Center (BRC) Design for Business Continuity Management," *Security by Design: Innovative perspectives on complex problems*, Advanced Sciences and Technologies for Security Applications A. J. Masys, ed., pp. 291-316, Cham: Springer, 2018.

3. H. J. Scholl, "Overwhelmed by Brute Force of Nature: First Response Management in the Wake of a Catastrophic Incident," *Electronic Government: Proceedings of the 18th IFIP WG 8.5 International Conference, EGOV 2019*, I. Lindgren et al., eds., Springer, 2019, pp. 105-124.

4. K. S. Cawley, "Assessing the Impact of Age and Experience on the Perceived Ease of Use of Crisis Information Management Software," 2020.

5. R. Prasanna, and T. J. Huggins, "Factors affecting the acceptance of information systems supporting emergency operations centres," *Computers in Human Behavior*, vol. 57, no. April, pp. 168-181, 2016/04/01/,

6. H. W. Fischer, "The Sociology of Disaster: Definitions, Research Questions, & Measurements Continuation of the Discussion in a Post-September 11 Environment," *International Journal of Mass Emergencies and Disasters*, vol. 21, no. 1, pp. 91-107, 2003.

7. Anonymous, *National Incident Management System*, FEMA P-501, FEMA, Washington, DC, 2008.

8. Anonymous, "National Response Framework," Homeland Security, ed., FEMA, 2013, p. iv/48.

9. J. V. Kelly, *FEMA's Initial Response to the Colorado Flood*, U.S. Department of Homeland Security, Office of Inspector General, Washington, DC, 2014.

10. S. F. McCauley, *FEMA Faces Challenges in Managing Information Technology*, Office of Inspector General, Department of Homeland Security, Washington, DC, 2015.

11. A. Delaney, and R. Kitchin, "Progress and prospects for data-driven coordinated management and emergency response: the case of Ireland," *Territory, Politics, Governance*, vol. [8], no. [6], pp. [1-16], 2020.
12. C. Son et al., "Investigating resilience in emergency management: An integrative review of literature," *Applied ergonomics*, vol. 87, no. n/a, pp. 1-16, 2020.

13. S. V. Hart, *Crisis Information Management Software (CIMS) Feature Comparison Report*, National Institute of Justice, Washington, DC, 2002.

14. C. Nikolai et al., "Leveraging WebEOC in support of the Haitian relief effort: Insights and lessons learned." pp. [1-5].

15. T. Li et al., "Data-driven techniques in disaster information management," *ACM Computing Surveys (CSUR)*, vol. 50, no. 1, pp. 1-45, 2017.

16. D. J. Barnett et al., "An analysis of after action reports from Texas hurricanes in 2005 and 2017," *Journal of public health management and practice*, vol. 27, no. 2, pp. E71-E78, 2021.

17. C. E. Sánchez, and L. D. Sánchez, "Case Study: Emergency Department Response to the Boston Marathon Bombing," *Operational and Medical Management of Explosive and Blast Incidents*, D. W. Callaway and J. L. Burstein, eds., pp. 363-367, Cham, CH: Springer, 2020.

18. C. Wukich, "More Monitoring, Less Coordination: Twitter and Facebook Use between Emergency Management Agencies," *Journal of Homeland Security and Emergency Management*, vol. 17, no. 3, pp. [1-29], 2020.

19. J. Levy, and R. Prizzia, "Building Effective Emergency Management Public-Private Partnerships (PPP) for Information Sharing," *Security by Design: Innovative perspectives on complex problems*, Advanced Sciences and Technologies for Security Applications A. J. Masys, ed., pp. 375-401, Cham: Springer, 2018.

20. S. Truptil et al., "Interoperability of information systems in crisis management: Crisis modeling and metamodeling," *Enterprise Interoperability III: New Challenges and Industrial Approaches*, K. Mertins et al., eds., pp. 583-594, London, UK: Springer, 2008.

21. S. K. Aros, and D. E. Gibbons, "Developing an agent-based simulation model of the use of different communication technologies in inter-organizational disaster response coordination." pp. 68-79.

22. C. Whelan, and A. Molnar, "Organising Across Boundaries: Communication, Coordination and Conflict," *Securing Mega-Events: Networks, strategies and tensions*, C. Whelan and A. Molnar, eds., pp. 91-122, London: Palgrave Macmillan, UK, 2018.

23. H. J. Scholl et al., "Informational Challenges in Early Disaster Response: The Massive Oso/SR530 Landslide 2014 as Case in Point," *Proceedings of the 50th Hawaii International Conference on System Sciences (HICSS-50)*, Waikoloa, Hawaii, 2017, pp. 2498-2508.

24. H. J. Scholl, K. Hubbel, and J. Leonard, "Communications and Technology Challenges to Situational Awareness: Insights from the CR16 Exercise." *Proceedings of the 1st ISCRAM Asia-Pacific Conference*, pp. 1-15. pp. 1-15, ISCRAM, (2018).

25. T. Kedia et al., "Technologies Enabling Situational Awareness During Disaster Response: A Systematic Review," *Disaster Medicine and Public Health Preparedness*, pp. 1-19, 2020.

26. S. Misra, P. Roberts, and M. Rhodes, "The ecology of emergency management work in the digital age," *Perspectives on Public Management and Governance*, vol. 3, no. 4, pp. 305-322, 2020.
27. J. Ritchie, J. Lewis, and E. Gillian, "Designing and selecting samples," *Qualitative research practice : a guide for social science students and researchers*, J. Ritchie and J. Lewis, eds., pp. 77-108, London : Thousand Oaks, Calif.: Sage Publications, 2003.

28. E. E. Holdeman, "Use of WebEOC is insanity defined," *Emergency Management: Disaster Zone*, August 27, Government Technology, 2020.

29. J. M. Corbin, and A. Strauss, "Grounded theory research: Procedures, canons, and evaluative criteria," *Qualitative sociology*, vol. 13, no. 1, pp. 3-21, 1990.

30. J. Fereday, and E. Muir-Cochrane, "Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development," *International journal of qualitative methods*, vol. 5, no. 1, 2006.

31. K. Charmaz, *Constructing grounded theory: A practical guide through qualitative analysis*, p.^pp. xxiii/207, London, UK: Sage Publications, 2006.

32. Anonymous, *National Incident Management System Supporting Evaluation Program (NIMS STEP): Guide*, Report, Federal Emergency Management Agency, Washington, DC, 2010.

33. H. J. Scholl, K. Hubbell, and J. G. Leonard, "Information Sharing and Situational Awareness: Insights from the Cascadia Rising Exercise of June 2016," *Proceedings of the 52nd Hawaii International Conference on System Sciences (HICSS-52)*, 2019, pp. 1-11.

34. H. J. Scholl, and B. J. Patin, "Resilient Information Infrastructures: Criticality and Role in Responding to Catastrophic Incidents," *Transforming Government: People, Process and Policy*, vol. 8, no. 1, pp. 28-48, 2014.

35. M. Turoff et al., "The design of a dynamic emergency response management information system (DERMIS)," *JITTA: Journal of Information Technology Theory and Application*, vol. 5, no. 4, pp. 1-2004.

36. Anonymous, *Incident Decision Support Software Application Note: System Assessment and Validation for Emergency Responders (SAVER)*, U.S. Department of Homeland Security, Washington, DC, 2010.