NIST GCR 15-1010

Critical Assessment of Existing 

Methodologies for Measuring or 

Representing Community Resilience of 

Social and Physical Systems

Francis M. Lavelle

Applied Research Associates, Inc.

Liesel A. Ritchie

University of Colorado – Boulder

Alexis Kwasinski

University of Pittsburgh

Brian Wolshon

Louisiana State University

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.GCR.15-1010

NIST GCR 15-1010

Critical Assessment of Existing Methodologies 

for Measuring or Representing Community 

Resilience of Social and Physical Systems

Prepared for

U.S. Department of Commerce

Engineering Laboratory

National Institute of Standards and Technology

Gaithersburg, MD 20899

By

Francis M. Lavelle

Applied Research Associates, Inc.

Liesel A. Ritchie

University of Colorado – Boulder

Alexis Kwasinski

University of Pittsburgh

Brian Wolshon

Louisiana State University

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.GCR.15-1010

December 2015

U.S. Department of Commerce

Penny Pritzker, Secretary

National Institute of Standards and Technology 

Willie May, Under Secretary of Commerce for Standards and Technology and Director 

i

Acknowledgments 

This report documents literature reviews and critical assessments conducted by the authors under the

Community Resilience Assessment Methodology task order of the Disaster and Failure Studies Program 

(NIST contract SB1341-12-CQ-0014, Task Order 14-373).

The NIST Project Manager and Technical Point of Contact for this Task Order was Dr. Erica Kuligowski. 

The authors would like to express our gratitude to Dr. Kuligowski for her guidance and feedback 

throughout the course of the project and for her thoughtful comments on this report. The authors also 

wish to thank Mr. Stephen Cauffman, Dr. Therese McAllister, and Ms. Nancy McNabb of the NIST 

Community Resilience Group and Mr. David Mizzen of Applied Research Associates for their valuable

inputs and comments throughout the project.
ii

This Page Intentionally Left Blank
iii

Contents

1. Introduction...........................................................................................................................................1

2. Critical Assessment Approach..............................................................................................................3

3. Assessment of SPUR Methodology......................................................................................................5

4. Assessment of Oregon Resilience Plan (ORP) Methodology...............................................................9

5. Assessment of UNISDR Disaster Resilience Scorecard for Cities.....................................................15

6. Assessment of the Community and Regional Resilience Institute (CARRI) Community Resilience 

System (CRS) .....................................................................................................................................21

7. Assessment of Communities Advancing Resilience Toolkit (CART)................................................27

8. Assessment of Baseline Resilience Indicators for Communities (BRIC)...........................................35

9. Assessment of Rockefeller Foundation City Resilience Framework (CRF) and City Resilience 

Index (CRI).........................................................................................................................................41

10. Assessment of NOAA Coastal Resilience Index................................................................................47

11. Assessment of FEMA Hazus Methodology........................................................................................55

12. Summary Assessment of Nine Existing Methodologies.....................................................................59

13. Concluding Remarks ..........................................................................................................................61

Appendices

A. Annotated Bibliography of Additional Community-Based Assessment Methodologies ...................63

B. Methodologies Considered for Electric Power Infrastructure Resilience...........................................71

C. Methodologies Considered for Information and Communication Infrastructure Resilience..............79

D. Methodologies Considered for Transportation Infrastructure Resilience...........................................85
iv

This Page Intentionally Left Blank
1

1. Introduction

This report documents literature reviews and critical assessments of existing methodologies that can be 

used to measure or assess community resilience. The work was conducted under Task 2 of the

Community Resilience Assessment Methodology (CRAM) task order of the National Institute of 

Standards and Technology’s (NIST) Disaster and Failure Studies Program.

Presidential Policy Directive PPD-21, dated 12 February 2013, defines resilience as follows:

The term “resilience” means the ability to prepare for and adapt to changing conditions and 

withstand and recovery rapidly from disruptions. Resilience includes the ability to withstand and 

recover from deliberate attacks, accidents, or naturally occurring threats or incidents.

The overall objective of the CRAM task order is to “provide a technical foundation for the firstgeneration methodology to assess resilience at the community scale.” The primary requirement for Task 2 

is to “deliver a critical assessment (interim report) of existing methodologies to measure or represent 

community resilience of social and physical systems.” 

A summary of the critical assessment approach is presented in Section 2, and detailed assessments of the 

nine existing methodologies are provided in Sections 3 through 11. Four methodologies specified in the 

task order statement of work are assessed in Sections 3 through 6: (3) the San Francisco Planning and 

Urban Research Association (SPUR) framework, (4) the Oregon Resilience Plan, (5) the United Nations

Disaster Resilience Scorecard and (6) the Community Resilience System. Five additional methodologies

selected by the project team in coordination with the NIST project manager are assessed in Sections 7

through 11: (7) the Communities Advancing Resilience Toolkit (CART), (8) Baseline Resilience 

Indicators for Risk (BRIC), (9) the Rockefeller Foundation’s City Resilience Framework and City 

Resilience Index, (10) the Coastal Resilience Index and (11) the Hazus Loss Estimation Methodology.

The critical assessment results for the nine existing methodologies are summarized in Section 12. 

The report concludes with four appendices. Appendix A provides an annotated bibliography of seven 

additional community-based assessment methodologies that were identified and considered during the 

course of this task. The final three appendices present sector-specific literature reviews in the areas of 

electric power infrastructure (Appendix B), information and communications infrastructure (Appendix C), 

and transportation infrastructure (Appendix D). The intent of the final three appendices is to summarize 

the information contained in these sources and highlight relative strengths and weaknesses of the sectorspecific methodologies relative to the NIST CRAM effort. Where appropriate, these summaries highlight

potentially useful and interesting ideas, information, resources that may be applicable within the context 

of general community planning. 
2

This Page Intentionally Left Blank
3

2. Critical Assessment Approach

Many community-wide resilience assessment methodologies have been proposed in the research 

literature. Nine existing methodologies representing ranges of community systems and modeling 

approaches are summarized and critically evaluated in Sections 3 through 11 of this report. The selected

methodologies were not necessarily developed specifically for the purpose of assessing community 

resilience, but they are all considered relevant and potentially applicable to the problem of community 

resilience assessment, either in whole or in part. Additional methodologies and improvements to existing 

methodologies are likely to emerge as further research and pilot studies are completed.

In reviewing the nine selected methodologies, the following dimensions and questions are considered:

1. Comprehensiveness

a. Is the methodology applicable to communities of varying sizes, demographics, social structures, 

and economies?

b. Is the methodology applicable across all hazards, or does it focus on specific hazards?

c. Does the methodology apply to a full range of recovery time scales or does it focus, for example, 

exclusively on short-term recovery issues?

d. What types of physical systems, social systems, and interdependencies can be represented in the 

methodology?

2. Utility

a. Is the methodology “user-friendly” (i.e., are the inputs relatively easy to obtain and are the 

outputs understandable to community leaders)?

b. Does the methodology require special technical resources or subject matter experts (SMEs)?

c. Do the outputs of the methodology provide useful/valuable information that supports community 

resilience planning and decision-making?

d. Does the methodology define and evaluate resilience in a manner that is consistent with PPD21?

3. Impacts Assessed

a. Does the methodology assess physical impacts to the built environment? Does it consider 

interdependencies within and between physical and social systems? Does it assess the time

needed to restore the functionality of the built environment following a disruptive event?

b. Does the methodology assess economic impacts? Does it assess the time needed to restore the 

economic activity to a level equal to or exceeding the pre-event level?

c. Does the methodology assess social impacts? Does it assess recovery times for social systems?

d. Does the methodology assess ecological impacts? Does it assess the time required for the 

environment to recover to conditions that are comparable to or superior to pre-event conditions?

4. Techniques Used – Which of the following assessment aids or techniques either must be used or can 

be used in the methodology?

a. Checklists

b. Interview or survey instruments

c. Rating systems (either numeric or qualitative)

d. Existing national datasets (e.g., demographic data, construction data, etc.)
4

e. Physical inspections or condition assessments of existing buildings and infrastructure systems

f. Engineering analysis or expert opinion

g. Statistical inference or statistical modeling techniques

h. Computer simulations

5. Critical Assessment

a. Maturity: Is the methodology still under development, or is it ready for use by communities in 

its current form? Have there been any significant revisions or improvements to the methodology 

since its initial version? Has it been used by others beyond the original developers?

b. Unique/Innovative: Are there significant aspects of the methodology that are unique, innovative, 

or otherwise noteworthy?

c. Objective and Repeatable: Are the outputs of the methodology heavily dependent on the 

individuals implementing it, or would other users with similar skills, training and resources be 

likely to produce similar outputs?

d. Scientific Merit:

i. Has the methodology been successfully demonstrated in a controlled setting?

ii. Has the methodology been successfully field tested?

iii. Have the methodology and the utility of its results been evaluated by the developers of the 

methodology? Have the assessment results been published?

iv. Has the methodology been successfully used to support real world planning decisions?

v. Has the methodology been independently evaluated? If so, in what settings?

vi. Is the process for implementing the methodology clearly defined? Can the outputs of the 

methodology be clearly understood by users?

vii. If the methodology combines or aggregates component- or system-level metrics into one or 

more community-level metrics, what is the theoretical or empirical basis for the algorithm?

e. Gaps: Have gaps of knowledge, data, or analytical approaches regarding resilience metrics been 

identified?

i. How can these gaps in knowledge, data or metrics be addressed?

ii. What are the next steps required to improve the methodology?

The above criteria and questions are evaluated in a narrative format in this study. No attempt is made to 

score or rank the methodologies. Rather, the assessment is intended to identify modeling strategies, 

features, and gaps in a manner that will support future efforts at NIST and elsewhere to develop robust 

community resilience assessment capabilities with a focus on improving the performance of the built 

environment in disruptive events and speeding up the recovery process to support community functions 

and restore services.
5

3. Assessment of SPUR Methodology

References:

1. http://www.spur.org/sites/default/files/publications_pdfs/SPUR_Seismic_Mitigation_Policies.pdf

2. http://www.spur.org/publications/spur-report/2009-02-01/defining-what-san-francisco-needs-itsseismic-mitigation

3. http://www.spur.org/featured-project/resilient-city

4. http://www.spur.org/spur-program/disaster-planning

5. http://www.spur.org/publications/spur-report/2009-02-01/dilemma-existing-buildings

6. http://www.spur.org/publications/spur-report/2009-02-01/lifelines

7. http://www.spur.org/publications/article/2010-07-06/transportation-and-rebuilding

Summary (Reference 1):

This paper addresses one aspect of the broader policy problem related to making San Francisco resilient 

in the face of a disaster - the standards we use for deciding when a structure is “safe enough.” Our 

building code embodies hundreds of judgment calls about how strong structures should be, but the public 

and the policy makers generally have no idea what these standards mean, what the outcomes will be from 

the “black box” of engineering decisions.

The truth is that when we choose our engineering standards we really are choosing to define how many 

deaths, how many building demolitions, and how long a recovery time we will have for various levels of 

earthquakes. Currently, the City of San Francisco has no adopted performance objectives for determining 

these factors. As a result,

 Design and construction requirements for new construction still focus mostly on preventing the 

loss of life and in most cases ignore the question of building damage and post-earthquake 

usability.

 Little is being done to rehabilitate older existing structures, which constitute the majority of 

buildings and which were built without earthquake-resistant features now required.

 There is no consistent approach to providing, maintaining, and restoring lifeline systems that are 

needed to support economic recovery.

The overall impact and cost of a disaster is strongly influenced by how long it takes to recover. The time 

needed to recover depends on the level of damage sustained by buildings, the availability of utilities, and 

how quickly communities can re-establish usable housing and livable environments.

This paper provides a new framework for improving San Francisco’s resilience through seismic 

mitigation policies. Our goals are to:

1. define the concept of “resilience” in the context of disaster planning,

2. establish performance goals for the “expected” earthquake that supports our definition of 

resilience,

3. define transparent performance measures that help us reach our performance goals; and

4. suggest next steps for San Francisco’s new buildings, existing buildings and lifelines.
6

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness

a. Applicability across communities of varying size and type

Focus is on the City and County of San Francisco (CCSF), but the approach is more 

broadly applicable.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc.

Focus is on the “expected” seismic event for San Francisco which is defined both 

probabilistically (10% probability of occurrence in 50 year, or 475-year mean recurrence 

interval) and with a specific scenario (magnitude 7.2 earthquake on the peninsula 

segment of the San Andreas Fault).

c. Applicability across different recovery time scales 

Focus is on three recovery phases: (1) Initial response (1-7 days), (2) Restoration of 

housing and utilities (30-60 days), and (3) Long term reconstruction (within 4 years).

d. Representativeness across different systems/interdependencies

Focus is on four main “clusters” of facilities and their supporting infrastructure: (1) 

critical response facilities, (2) emergency housing, (3) neighborhoods (essential city 

service facilities, housing, schools, medical offices, neighborhood retail), and (4) 

community recovery.

2. Utility

a. User-friendliness – The SPUR approach provides a framework presenting current status 

and desired “target states” in a format that is accessible to community planners and 

decision-makers.

b. Level of technical resources required to implement the methodology – The SPUR 

methodology requires seismic engineering expertise to define the “expected event” and 

its likely consequences.

c. Value of methodology outputs in supporting resilience planning – By emphasizing 

restoration times of broad “clusters” of buildings and infrastructure, the SPUR 

methodology seems to provide a useful framework for community-level awareness and 

decision making.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? –

Yes, with a focus on withstanding and recovering from the expected seismic event for 

CCSF.

3. Ability to assess/measure/predict...

a. Physical impacts, interdependencies, and recovery times – The methodology directly 

assesses physical impacts and recovery times for the expected seismic event. The 

importance of interdependencies, comprehensive planning, and establishing priorities for 

lifeline mitigation are highlighted in the SPUR Lifelines report (Reference 6).

b. Economic impacts and recovery times – The methodology does not explicitly seek to 

model economic impacts, but much of the analysis and/or expert opinion required to 

assess physical impacts, interdependencies and recovery times could be used as inputs to 

economic models.
7

c. Social impacts and recovery times – The methodology does not explicitly seek to model 

social impacts, but much of the analysis and/or expert opinion required to assess physical 

impacts, interdependencies and recovery times could be used as inputs to social models.

d. Ecological impacts and recovery times – No

4. Which of the following techniques are used in the methodology?

a. Checklists

b. Interviews

c. Ratings – Main output is a matrix of current and target recovery times for each of four 

main “clusters” of facilities and supporting infrastructure

d. Physical inspections

e. Exercises (e.g., tabletop)

f. Engineering analyses – Required to assess current status

g. Statistical inference

h. Simulations

5. Critical Assessment

a. Maturity – Published in 2009. Adapted and built upon by Oregon Resilience Plan and the 

NIST Community Resilience Planning Guide.

b. Uniqueness/innovativeness – Focus on recovery times as primary performance goal 

rather than life safety or economic impacts is innovative.

c. Objective and repeatable? – The SPUR methodology was developed to support a specific 

set of planning objectives for the City and County of San Francisco. The procedures

developed for assessing the current status of the community and determining whether 

desired recovery times are achievable are systematic and rational, but generalizing and 

standardizing the process into a step-by-step methodology for general use for a broad 

range of communities and hazards was beyond the scope of the SPUR effort.

d. Scientific basis/merit/empirical evidence to support the method

i. Current/prior use of methodology – San Francisco, Oregon

ii. Extent to which the approach been field tested – Unknown

iii. Extent to which the approach has been self-assessed – Unknown

iv. Has it been applied to real-world planning and management? – Yes

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? – Unknown

vi. Clarity – The general approach is clear, but the details of how the current 

performance levels and expected recovery times were assessed are not 

documented in the references listed at the start of this section.

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? – The methodology 

focuses on the expected performance and recovery times for four main “clusters” 

of facilities and their supporting infrastructure (see Question 1.d).
8

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? – Generalize the 

SPUR approach to other communities and hazards and develop an objective and 

repeatable step-by-step approach for assessing performance and recovery times. 

Expand the methodology to explicitly consider economic, social, and ecological 

impacts.

ii. What are the next steps that must be taken to improve the methodologies 

currently available? – Develop objective methodologies for assessing expected 

performance and recovery times by “cluster” for both current and proposed states 

of the community.
9

4. Assessment of Oregon Resilience Plan (ORP) Methodology

References:

1. http://www.oregon.gov/OMD/OEM/osspac/docs/Oregon_Resilience_Plan_Final.pdf

2. http://www.oregon.gov/OMD/OEM/osspac/docs/Oregon_Resilience_Plan_Executive_Summary_

Final.pdf

3. http://www.oregon.gov/OMD/OEM/Pages/osspac/osspac.aspx#Oregon_Resilience_Plan

4. http://www.oregon.gov/OMD/OEM/osspac/docs/or_resilience_planning.pdf

Summary (Reference 1):

House Resolution 3, adopted in April 2011, directed the Oregon Seismic Safety Policy Advisory 

Commission (OSSPAC) “to lead and coordinate preparation of an Oregon Resilience Plan that reviews 

policy options, summarizes relevant reports and studies by state agencies, and makes recommendations 

on policy direction to protect lives and keep commerce flowing during and after a Cascadia earthquake 

and tsunami.” OSSPAC assembled eight task groups, comprising volunteer subject-matter experts from 

government, universities, the private sector, and the general public. An Advisory Group of public- and 

private-sector leaders oversaw the Task Groups’ work, assembled in the portfolio of chapters that make 

up the plan.

OSSPAC offered the following definition of the seismic resilience goal:

“Oregon citizens will not only be protected from life-threatening physical harm, but because of 

risk reduction measures and pre-disaster planning, communities will recover more quickly and 

with less continuing vulnerability following a Cascadia subduction zone earthquake and 

tsunami.”

Each group was charged with three tasks for four affected zones (tsunami, coastal/earthquake only, 

valley, and central/eastern Oregon):

1. Determine the likely impacts of a magnitude 9.0 Cascadia earthquake and tsunami on its 

assigned sector, and estimate the time required to restore functions in that sector if the 

earthquake were to strike under present conditions;

2. Define acceptable timeframes to restore functions after a future Cascadia earthquake to fulfill 

expected resilient performance; and 

3. Recommend changes in practice and policies that, if implemented during the next 50 years, will 

allow Oregon to reach the desired resilience targets.

The purpose of the analysis is to identify steps needed to eliminate the gap separating current 

performance from resilient performance, and to initiate that work through capital investment, new 

incentives, and policy changes so that the inevitable natural disaster of a Cascadia earthquake and 

tsunami will not deliver a catastrophic blow to Oregon’s economy and communities.

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness

a. Applicability across communities of varying size and type

Somewhat. The ORP is mostly focused on the particular setting and conditions found in 

Oregon for a Cascadia Subduction Zone earthquake as mandated by the State of Oregon 

Legislature.
10

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc.

Somewhat. The ORP is mostly focused on the particular setting and conditions found in 

Oregon for a Cascadia Subduction Zone earthquake.

c. Applicability across different recovery time scales

Somewhat. The ORP is mostly focused on the particular setting and conditions found in 

Oregon for a Cascadia Subduction Zone earthquake. Within that context, the ORP 

addresses the time frames required to restore functions across seven sectors: business and 

workforce continuity, communities, critical and essential buildings, transportation, 

energy, information and communications, and water and wastewater systems.

d. Representativeness across different systems/interdependencies

The ORP is very comprehensive in considering physical systems because one of the goals 

of the ORP as mandated by the State of Oregon’s Legislature is to identify infrastructure 

investment needs to increase resilience. Another goal of the plan is to “keep commerce 

flowing.” Therefore, a portion of the plan is dedicated to social systems, with a focus on 

economic/business systems, such as a discussion on “business workforce 

interdependency.” Other social systems are also considered with respect to their 

relationship with economic/business systems, for example, by acknowledging the 

importance of the education system not only through its primary function but also as a 

support system for emergency response and temporary sheltering/housing and for 

business recovery. Hence, the ORP considers physical systems and some social systems, 

including a recognition of their interdependencies.

2. Utility

a. User-friendliness. The target audience is state and local planners and decision-makers.

b. Level of technical resources required to implement the methodology. Extensive. 

Particularly in terms of experts in each field. Based on SPUR.

c. Value of methodology outputs in supporting resilience planning. High. The main goal of 

the ORP is to translate observations into planning.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? 

The ORP is quite consistent with PPD-21 within the specific context of a Cascadia 

Subduction Zone earthquake affecting the State of Oregon. It tends to be oriented towards 

preparation and recovery, but it is also acknowledged that mitigating or preventing 

damage, where possible, clearly implies a more rapid recovery. 

3. Ability to assess/measure/predict... 

a. Physical impacts, interdependencies, and recovery times. Yes, these are primary 

considerations and are covered in detail in Chapters 3-8 of the ORP.

b. Economic impacts and recovery times. Direct economic impacts and the influence and 

interdependencies of physical and social systems associated with economic and business 

activities are discussed in Chapter 2 of the ORP.

c. Social impacts and recovery times. Some aspects of social impacts in coastal 

communities vulnerable to tsunamis are discussed in Chapter 3 of the ORP. Specific 

issues addressed include land use planning; preparedness, evacuation, and relief efforts; 

and restoration of services in coastal communities.
11

d. Ecological impacts and recovery times. The primary focus of the ORP is on the built 

environment, but sustainable development and protection of the natural environment 

against impacts resulting from damage to the built environment are discussed at various 

points throughout the plan.

4. Which of the following techniques are used in the methodology?

a. Checklists

b. Interviews

c. Ratings – Yes

d. Physical inspections

e. Exercises (e.g., tabletop)

f. Engineering analyses

g. Statistical inference

h. Simulations

i. Other – Comparison with previous events and subject matter experts assessments

5. Critical Assessment

a. Maturity. Substantial. The ORP was completed in February 2013 and submitted to the 

legislature, which is considering implementation of some recommendations provided in 

the plan. 

b. Uniqueness/innovativeness. Somewhat, as the methodology is similar to that 

implemented by SPUR based on target resilience objectives for various infrastructure

systems. As a statewide plan, the ORP differs from SPUR in terms of the geographic 

scope of the assessment.

c. Objective and repeatable? The ORP methodology was developed to support a specific set 

of planning objectives for the State of Oregon. The process developed for assessing 

expected impacts and recovery times is systematic and rational, but its reliance on subject 

matter expert (SME) assessments of current and targeted recovery times would be 

difficult to independently replicate. Generalizing and standardizing the process into a 

step-by-step methodology for general use for a broad range of communities and hazards 

was beyond the scope of the ORP effort.

d. Scientific basis/merit/empirical evidence to support the method. The method follows an 

approach based on tables similar to that implemented by SPUR. However, the goal of 

identifying system performance goals led to preparing such tables based on SME 

assessments and evaluation of community needs. Thus, there are no statistical analyses or 

mathematical equations or relationships to support the tables presented in the ORP plan.

As such, the method is seen more oriented towards representing target resilience goals 

instead of assessing expected resilience. This method is justified as the mandate from the 

Oregon Legislature was to “Define acceptable timeframes to restore functions after a 

future Cascadia earthquake” rather than to develop a detailed resilience model for the 

built environment.

i. Current/prior use of methodology. The approach is a standard one in the sense 

that the methodology follows a mandate to identify potential resilience gaps in 

order to decide on future infrastructure investments and mitigate the effects of a 

future Cascadia Subduction Zone earthquake. That is, the use of the methodology 
12

follows the conventional approach of identifying risks and resilience deficiencies, 

evaluating primarily physical infrastructure issues contributing to higher risks or 

lower potential resilience, and planning investment needs in order to mitigate the 

hazard’s effect and improve resilience.

ii. Extent to which the approach been field tested. Not applicable.

iii. Extent to which the approach has been self-assessed. Not applicable.

iv. Has it been applied to real-world planning and management? Yes

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? No

vi. Clarity. The process developed for assessing expected impacts and recovery 

times is systematic and rational, but its reliance on subject matter expert (SME) 

assessments of current and targeted recovery times would be difficult to 

independently replicate.

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? Yes, the ORP uses the 

concept of the resiliency triangle as the system level metric, but it is not clear 

how the triangle components are combined in order to reach the community-level 

metric. A copy of the triangle is included below. The concept of the triangle is 

that more resilient systems or communities will show triangles with a smaller 

area as the disruption level (vertical axis) and recovery time (horizontal axis) are 

expected to be less for more resilient systems.

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? Populate the tables 

based on computer-based simulations running models valid for different 

scenarios/disasters.

ii. What are the next steps that must be taken to improve the methodologies 

currently available? Consider how human processes fit as components of social 

and physical systems and how their implementation can be modeled as part of a 

system simulation. Develop improved resiliency models for social and physical 

systems which includes the identification of resiliency metrics. Improve the 
13

understanding of dependencies between social and physical systems, and two 

social systems, and two physical systems. Develop an approach that would 

combine social system resiliency metrics with physical systems resiliency 

metrics into a single community resiliency metric.
14

This Page Intentionally Left Blank
15

5. Assessment of UNISDR Disaster Resilience Scorecard for Cities

References:

1. http://www.unisdr.org/2014/campaign-cities/Resilience%20Scorecard%20V1.5.pdf

2. http://www.unisdr.org/2014/campaigncities/Scorecard%20FAQs%20March%2010th%202014.pdf 

3. http://www.unisdr.org/campaign/resilientcities/home/toolkitblkitem/?id=3

Summary (Reference 1):

This scorecard provides a set of assessments that will allow cities to understand how resilient they are to 

natural disasters. It is based on the UNISDR’s “Ten Essentials” of disaster management and adds 

significant additional detail and quantification beneath the UNISDR’s Local Government Self-Assessment 

Tool (LGSAT). It has been compiled by IBM and AECOM, who are members of UNISDR’s Private Sector 

Advisory Group (PSAG).

The term “resilience” is often taken to include responses to a spectrum of factors, ranging from 

“chronic” stresses such as environmental pollution, ground water depletion or deforestation, to “acute” 

stresses such as floods, droughts, earthquakes, hurricanes or wild-fires. “Disaster resilience” as defined 

here is at the “acute” end of this spectrum: it covers the ability of a city to understand the disaster risks it 

may face; to mitigate those risks; and to respond to disasters that may occur, in such a way as to 

minimize loss of or damage to life, livelihoods, property, infrastructure, economic activity and the 

environment. Clearly, disaster resilience will be affected by the chronic stresses that the city may also 

face, for example where deforestation increases the propensity for flash flooding, or where water 

pollution exacerbates the impact of a drought.

The Disaster Resilience Scorecard (hereafter, “the Scorecard”) is intended to enable cities to establish a 

baseline measurement of their current level of disaster resilience, to identify priorities for investment and 

action, and to track their progress in improving their disaster resilience over time. It consists of 85 

disaster resilience evaluation criteria and focuses on the following aspects:

 Research, including evidence-based compilation and communication of threats and needed 

responses

 Organization, including policy, planning, coordination and financing

 Infrastructure, including critical and social infrastructure and systems and appropriate 

development

 Response capability, including information provision and enhancing capacity

 Environment, including maintaining and enhancing ecosystem services

 Recovery, including triage, support services and scenario planning.

Each evaluation criterion is broken down to set out the aspect of disaster resilience being measured, an 

indicative measurement and the measurement scale (from 0 to 5, where 5 is best practice).

The scorecard provides an aspirational definition of disaster resilience – it is very unlikely that any city 

would currently score maximum points, and most will not score more than 50%. Its intention is to guide 

cities towards optimal disaster resilience, and to challenge complacency. This demanding standard 

reminds cities that there is always more that could be done, and to establish investment goals (including 

time and effort) for achievement over a period of years.
16

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness

a. Applicability across communities of varying size and type

Focus is on cities, but the basic approach appears to be adaptable to communities of all 

sizes. Mentions 2,220 participating cities worldwide. Also mentions vertical integration 

of local plans in regional framework.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc.

Focus appears to be on natural hazards, but it is not limited to specific hazard types. 

c. Applicability across different recovery time scales

Not readily apparent. The assessment questions are oriented toward conditions and 

system characteristics as they presently exist.

d. Representativeness across different systems/interdependencies

Notes the need to be inclusive and the need for multi-stakeholder involvement

e. Other issues or ideas addressed:

 Integration into routine planning

 Legal considerations

 Early warning capabilities

 Reducing underlying threat/risk factors

2. Utility

a. User-friendliness

Appears to be user-friendly as a website. Clickable links, easy to follow.

b. Level of technical resources required to implement the methodology

The level of technical resources needed to properly conduct the evaluation could be 

significant given the breadth of the 85 questions and level of detail required to properly 

score each response. The instructions state the “full completion of the scorecard is likely 

to require a number of days, even if all the required information is to [sic] hand.” As with 

any other assessment, the Scorecard instructions also properly point out that the user 

“will need a clear understanding of the risk of each possible disaster and its impacts on 

your city. If you do not have these risk scenarios defined, it will not be possible to 

complete the scorecard, and your first step must therefore be to create them.”

c. Value of methodology outputs in supporting resilience planning

Would have significant value to identify what areas are lacking.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21?

Yes. The Scorecard methodology states “The Disaster Resilience Scorecard consists of 

85 separate assessments (questions), each designed to rate on a 0 to 5 scale an aspect of a 

city’s preparedness for, and ability to recover from, natural hazards.” Although the 

methodology specifically mentions natural hazards, the information appears to be very 

flexible and adaptable to virtually any hazard/threat condition. The intent is definitely 

there.
17

3. Ability to assess/measure/predict...

a. Physical impacts, interdependencies, and recovery times

The description of the methodology suggests that it has an ability to predict vulnerability 

of physical systems at a high level. However, there was not any clear evidence that the 

outcomes of the evaluation technique could be used to assess system interdependencies or 

recovery times.

b. Economic impacts and recovery times

Again, this is discussed prominently, but not the specifics of how or what. The 

methodology mostly provides suggestions of what to check in the areas of economic

impacts and financial services.

c. Social impacts and recovery times

Specifically mentions hospitals, schools, shelters, and vulnerable populations. Also 

includes suggested assessment of “psycho-social (psychological, emotional) impacts of 

disasters”

d. Ecological impacts and recovery times

Yes for impacts, but recovery times are not explicitly discussed. Specific aspects 

addressed include:

 coastal zones

 wetlands

 water resources

 river basins

 fisheries

4. Which of the following techniques are used in the methodology?

a. Checklists – Yes. It includes 85 assessment items structured in 10 sections, corresponding 

to the UNISDR’s “Ten Essentials”:

 Essential 1: Put in place organization and coordination to understand and reduce 

disaster risk, based on participation of citizen groups and civil society. Build local 

alliances. Ensure that all departments understand their role to disaster risk reduction 

and preparedness.

 Essential 2: Assign a budget for disaster risk reduction and provide incentives for 

homeowners, low-income families, communities, businesses and public sector to 

invest in reducing the risks they face.

 Essential 3: Maintain up-to-date data on hazards and vulnerabilities, prepare risk 

assessments and use these as the basis for urban development plans and decisions. 

Ensure that this information and the plans for your city's resilience are readily 

available to the public and fully discussed with them.

 Essential 4: Invest in and maintain critical infrastructure that reduces risk, such as 

flood drainage, adjusted where needed to cope with climate change.

 Essential 5: Assess the safety of all schools and health facilities and upgrade these as 

necessary.
18

 Essential 6: Apply and enforce realistic, risk compliant building regulations and 

land use planning principles. Identify safe land for low-income citizens and develop 

upgrading of informal settlements, wherever feasible.

 Essential 7: Ensure education programmes and training on disaster risk reduction 

are in place in schools and local communities.

 Essential 8: Protect ecosystems and natural buffers to mitigate floods, storm surges 

and other hazards to which your city may be vulnerable. Adapt to climate change by 

building on good risk reduction practices.

 Essential 9: Install early warning systems and emergency management capacities in 

your city and hold regular public preparedness drills.

 Essential 10: After any disaster, ensure that the needs of the survivors are placed at 

the centre of reconstruction with support for them and their community organizations 

to design and help implement responses, including rebuilding homes and livelihoods.

b. Interviews

c. Ratings – Yes, a 5 point scale used to self-assess the status and level of progress on each 

of the 85 assessment items. The stated intent of the Scorecard (Reference 2) is to

provide a single integrated perspective on a city’s total disaster resilience posture, 

and on the connections between the many different aspects of disaster resilience, 

while also identifying gaps in plans and provisions. The resulting information can 

then also be used for:

 Improving the Disaster Resilience Scorecard for use by other cities, drawing on 

your experience of piloting it.

 Improving the UN’s Hyogo Framework for Action (HFA), of which the 2nd

iteration is due in 2015.

 Improving the current “Ten Essentials” for Making Cities Resilient prescribed 

by UNISDR

It is intended that the Disaster Resilience Scorecard will be useful for the day-to-day 

business of the city.

d. Physical inspections

Inspections are mentioned, but details are not provided.

e. Exercises (e.g., tabletop)

Yes. Training programs for responders are mentioned as are drills/education/outreach for 

the public.

f. Engineering analyses

There is definitely discussion of the need to do this, but the assessment does not detail 

how these should/would/could be carried out.

g. Statistical inference

h. Simulations
19

5. Critical Assessment

a. Maturity

Inconclusive. The first city to apply the Scorecard, Coimbatore, India, was announced in 

June 2014.

b. Uniqueness/innovativeness

The Scorecard includes interesting links to city-to-city learning and “role models”.

They include a local government self-assessment tool (Reference 3) with the purposes to:

 Help local governments engage with different stakeholders to map and 

understand existing gaps and challenges in disaster risk reduction in their city or 

locality.

 Set a baseline and develop status reports for cities and municipalities that have 

committed to the Making Cities Resilient Campaign and its Ten Essentials.

 Complement information gathered through the national Hyogo Framework for 

Action monitoring system (HFA Monitor) by providing local-level information. 

Cities can choose to share their results with national HFA focal points as part of 

the national reporting process.

c. Objective and repeatable?

Somewhat. The definitions of the 5 point rating scales for each of the 85 assessment

items are fairly detailed, but still open to interpretation. The Scorecard itself states: 

“While the scorecard aims to be systematic, individual scores are often unavoidably 

subjective.”

d. Scientific basis/merit/empirical evidence to support the method

i. Current/prior use of methodology

Extent not known, but given the amount of participating communities listed, 

there appears to be significant potential for usage.

ii. Extent to which the approach been field tested

See above

iii. Extent to which the approach has been self-assessed

Not apparent from available information

iv. Has it been applied to real-world planning and management?

Appears to have been, although extent to which is not readily clear

v. Does evaluative information exist regarding how the methodology has worked in 

different settings?

Not apparent from available information

vi. Clarity

Very clear

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined?
20

It appears that the current approach is to compute any overall score as the 

percentage of 425 possible points (85 x 5 = 425). There is no readily apparent 

theoretical or empirical basis provided. Presumably, the number of questions 

allocated to each of the “Ten Essentials” represents an implicit weighting of the 

relative importance of each Essential in the view of the Scorecard developers.

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? Establish the validity 

of the scoring criteria for the individual questions and the validity of the resulting 

overall score. 

ii. What are the next steps that must be taken to improve the methodologies 

currently available? Demonstrate that the Scorecard can be used not only for 

assessing the current level of resilience in a community, but also for effectively 

assessing different community resilience improvement strategies.
21

6. Assessment of the Community and Regional Resilience Institute

(CARRI) Community Resilience System (CRS)

References:

1. http://www.resilientus.org/recent-work/community-resilience-system/

2. http://www.resilientus.org/wp-content/uploads/2013/03/CRSFAQs_CARRI_website_1321021479.pdf

3. http://www.resilientus.org/wp-content/uploads/2013/05/CRS-Final-Report.pdf

Summary (Reference 2):

The Community Resilience System (CRS) is an action-oriented, web-enabled process that helps 

communities to assess, measure, and improve their resilience to the variety for [sic] threats and 

disruptions of all kinds, and ultimately be rewarded for their efforts. 

The CRS brings together people, process and technology to improve resilience in individual communities. 

The system includes not only a knowledge base to help inform communities on their resilience path but 

also a process guide that provides a systematic approach to moving from interest and analysis to 

visioning and action planning. It also provides a collaborative mechanism for other interested 

stakeholders to support community efforts.

The CRS was developed by the Community Resilience System Initiative (CRSI). The CRSI was a 15-month 

collaborative process charged with determining what American communities need in order to become 

more resilient to the variety of threats they face and recommending a concrete course of action that will 

support communities in their resilience-building efforts. CRSI involved more than 150 practitioners and 

researchers from diverse sectors and disciplines who worked in groups to help inform the development of 

the Community Resilience System (CRS).

Additional notes:

 This was a DHS/FEMA funded initiative

 Began in 2010, convening three working groups: researchers (the Subject Matter Group), 

community leaders (the Community Leaders Group), and government/private sector 

representatives (the Resilience Benefits Group)

 The findings of these working groups culminated in the development of the CRS web-based tool 

and a pilot implementation commencing in the summer of 2011

 CARRI conducted a limited evaluation of the pilot process; the information presented beginning 

on page 53 (see also Appendix B) of the CRS Final Report is useful feedback about challenges as 

well as strengths of the system that may be applicable for implementation of other communityfocused systems

 According to the report, the process is most effective when facilitated by those experienced with 

the CRS – i.e., CARRI (or potentially those trained in its use)

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness

a. Applicability across communities of varying size and type – The methodology appears to 

be applicable to communities of all sizes and types.
22

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc. – The CRS is not hazard-specific; communities are able to select issues 

and hazards that are of concern to them. Thus, it is broadly applicable.

c. Applicability across different recovery time scales – Unclear from the available materials. 

It seems that each community has the flexibility to determine this.

d. Representativeness across different systems/interdependencies – The CRS includes the 

following 18 distinct Community Service Areas (CSAs): 

1. Arts, Entertainment, and Recreation

2. Communications

3. Community Records

4. Economic

5. Education

6. Energy

7. Financial

8. Food Supply and Distribution

9. Housing

10. Individuals and Families

11. Local Government

12. Natural Environment

13. Public Health

14. Public Safety and Security

15. Solid Waste

16. Transportation

17. Water

18. Workforce

The CRS assessment process is designed to help communities recognize critical 

interdependencies between these 18 distinct CSAs. Assessors are prompted to identify 

critical assets and the “provider” of those assets.

2. Utility

a. User-friendliness – The CRS is designed specifically for use by community leaders and it 

seems that the system itself has a strong user-friendly approach. The limited evaluation 

information provided about the CRS (based on observations of the CARRI team and of a 

survey they sent out to pilot participants) suggests that there are some things that need to 

be worked out before the CRS could be used broadly. Also, the CARRI team notes that a 

facilitated approach (i.e., an outside group coming in, such as CARRI), is most effective. 

“The CRS process works more productively as a “partially facilitated” model where some 

supportive expertise assists communities in applying aspects of resilience to and 

embedding them within their community circumstances and processes.”
23

b. Level of technical resources required to implement the methodology – The system seems 

to rely heavily on available resources in a given community—whatever those are. The 

more robust the local knowledge/capacity/infrastructure, the more sophisticated the 

result.

c. Value of methodology outputs in supporting resilience planning – The methodological 

approach seems very robust and involves six phases: engagement, assessment, visioning, 

planning, implementing, and monitoring and maintaining.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? – 

Yes. Although developed with a focus on PPD-8 and the National Recovery Framework, 

the language is similar: 

“Community resilience is the ability of a community to anticipate risk, limit impact, and 

bounce back rapidly through adaptation, evolution, and growth in the face of turbulent 

change.”

“When a community is truly resilient, it should be able to avoid the cascading system 

failures to help minimize any disaster's disruption to everyday life and the local economy. 

A resilient community is not only prepared to help prevent or minimize the loss or 

damage to life, property and the environment, but also it has the ability to quickly return 

citizens to work, reopen businesses, and restore other essential services needed for a full 

and swift economic recovery.”

3. Ability to assess/measure/predict...

a. Physical impacts, interdependencies, and recovery times –

b. Economic impacts and recovery times –

c. Social impacts and recovery times –

d. Ecological impacts and recovery times

For each of the areas above (a-d), communities choose which CSAs they will assess, and they 

may choose to evaluate only a subset of the CSAs at a time. Further, the CRS design assumes that 

most communities will choose Assessors who are subject matter experts to complete assessments 

of the individual CSAs. Typically, the Assessors will only answer specific questions for one or 

two community services. Questions are yes/no in format and are organized in 3 categories for 

each CSA: community capacity, critical assets, and recovery resources.

The questions are tailored for each of the CSAs. The answer to questions may trigger additional 

questions. For many of the questions, comment fields are provided so that communities may 

answer the questions as specifically as possible; this feature increases the effectiveness of the 

Potential Actions that are generated at the close of the Assessment since the specific comments 

associated with the questions also appear with the Potential Actions.

Once the community has answered questions regarding capacity for a specific CSA, assessors 

must next identify Critical Assets for each CSA they have chosen to evaluate. This section of the 

Assessment is designed to enable communities to think explicitly about the assets in their 

community that are critical to their ability to provide the functions and services of a specific 

CSA. Assessors are asked to identify not only the “asset” itself, but to also explicitly identify the 

“provider” of the asset. This combination of explicitly identifying assets and providers is 

designed to assist communities in recognition of dependencies and potential interdependencies 

within their community and also to help communities recognize when they are depending on 

assets that are provided from outside the community – thus, introducing potential complexities in 

their ability to control the “readiness” and resilience of specific assets upon which they depend.
24

Once assets and providers are input for a specific CSA, Assessors are guided through a series of 

questions that help assess the vulnerability of each critical asset. As with other sections of the 

Assessment, the questions answered here are tailored by the significant threats identified in the 

Threat Assessment worksheet.

Following the Critical Assets questions, Assessors are guided through an analysis of the Recovery 

Resources at the community’s disposal. As with Community Capacity and Critical Assets 

sections, the Resource Recovery questions are specific to each CSA and are driven by the 

community’s significant threats. These questions help the community assess the means at their 

disposal to rapidly recover the functions and services of the CSAs they are evaluating.

4. Which of the following techniques are used in the methodology? 

a. Checklists

b. Interviews

c. Ratings

d. Physical inspections

e. Exercises (e.g., tabletop)

f. Engineering analyses 

g. Statistical inference

h. Simulations

This will vary by community. The web process is a checklist-driven approach, but the 

information input into the Community Scorecard could apparently involve any of the above.

5. Critical Assessment

a. Maturity – Report published in 2013 with pilot and evaluation information. There were 8 

communities of differing size and type involved in the pilot: Annapolis/Anne Arundel 

County, MD; Anaheim, CA; Charleston Tri-Counties Region, SC; Gadsden, AL; 

Greenwich, CT; Gulfport, MS; Mt. Juliet, TN, and St. Louis/St. Louis County, MO. A 

slimmed down version of the process has also been used recently in Park City, UT. Two 

of the pilot communities dropped out of the initiative shortly after they joined.

b. Uniqueness/innovativeness – The web-based, dynamic approach is innovative. It is also 

innovative in that CARRI has attempted to implement the process in a variety of sites 

with communities of different sizes.

c. Objective and repeatable? – The methodology is replicable. The way communities will 

go about their assessment activities is not standardized; this system relies on local subject 

matter expertise, which will likely produce differing assessment approaches (in terms of 

what gets input into the system) and thus non-standardized results.

d. Scientific basis/merit/empirical evidence to support the method

i. Current/prior use of methodology – Piloted in 8 different communities (although 

2 dropped out) and partially implemented in another

ii. Extent to which the approach been field tested – Piloted in 8 different 

communities (although 2 dropped out) and partially implemented in another

iii. Extent to which the approach has been self-assessed – Good documentation of 

evaluation findings – although limited in scope
25

iv. Has it been applied to real-world planning and management? – Yes

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? – Yes. Evaluation feedback is included in the final report

beginning on page 53 and also in Appendix B.

http://www.resilientus.org/wp-content/uploads/2013/05/CRS-Final-Report.pdf

vi. Clarity – The process itself is clear; what is not entirely clear is what the 

products/outcomes of the process are. This is due in part to the fact that the CRS 

was only piloted, and apparently not carried to completion in any of the 

communities.

According to a response from CARRI: 

“To answer your most important question, we saw (and still see) the 

most important end product as action. Where communities have taken 

action, we judge our involvement to have been successful.” 

“Major resilience enhancement actions that resulted were development of 

a “Recovery Communications Network” in the Charleston Tri-County 

area, and a revitalized “Hi, Neighbor” campaign in Anaheim that 

included the business sector.”

Other notes: See beginning on page 36 of this report for progress in each of the 

pilot communities

http://www.resilientus.org/wp-content/uploads/2013/05/CRS-Final-Report.pdf

DHS/FEMA asked CARRI to adapt the CRS to institutions of higher education, 

which diverted resources and hindered follow-up efforts in the pilot 

communities.

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? – N/A

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? The list of metrics 

(community service areas) is robust. It depends on how communities decide to 

approach using the system and whether they have the necessary technical 

expertise and resources to delve deeply into the metrics as they go through the 

process.

ii. What are the next steps that must be taken to improve the methodologies 

currently available? The process associated with the approach is robust—it has 

been tried and there is feedback on what seems to work, and what does not. For 

the purposes of NIST’s Community Resilience Planning Guide and the focus on 

the built infrastructure, the CARRI system would need to address 

interdependencies and system.
26

This Page Intentionally Left Blank
27

7. Assessment of Communities Advancing Resilience Toolkit (CART)

References:

1. http://www.oumedicine.com/docs/ad-psychiatry-workfiles/cart_onlinefinal_042012.pdf?sfvrsn=2

2. Pfefferbaum, R.L., Pfefferbaum, B., Nitiema, P., Houston, J.B., and Van Horn, R.L., “Assessing 

Community Resilience: An Application of the Expanded CART Survey Instrument With 

Affiliated Volunteer Responders,” American Behavioral Scientist, September 24, 2015, retrieved 

at http://abs.sagepub.com/content/early/2014/09/24/0002764214550295.full.pdf+html

Summary:

This report presents the results of a toolkit developed as part of work conducted by the Terrorism and 

Disaster Center at the University of Oklahoma Health Sciences Center. It was funded by the Substance 

Abuse and Mental Health Services Administration, U.S. Department of Health and Human Services, and 

the National Consortium for the Study of Terrorism and Responses to Terrorism, U.S. Department of 

Homeland Security, and by the Centers for Disease Control and Prevention.

This approach was designed to enhance community resilience through planning and action. It engages 

community organizations in collecting and using assessment data to develop and implement strategies for 

building community resilience with respect to disaster prevention, preparedness, response, and recovery. 

Using a combination of qualitative and quantitative approaches, CART is based on the following three 

key aspects of group behavior:

 Communication among group members permits the group to pool skills and knowledge to achieve

better outcomes,

 Members can learn and grow as a result of group interactions, and

 Group participation can facilitate acceptance and implementation of group goals

The CART process involves the following steps:

1. Generating a community profile (CART Team and Partners)

2. Refine the community profile (Community Work Groups)

3. Develop a strategic plan (Community Planning Groups)

4. Implement the plan (Community Leaders and Groups)

Additional notes:

 The authors report that there have been some early survey applications, including a sample of 

Community Emergency Response Team (CERT) members in Arizona, two community samples, 

and a university-wide survey sample. The authors are preparing manuscripts describing the early 

community samples and plan to report on these studies in the near future.

 One article was found that describes CART’s implementation with a convenience sample of 64 

study participants who were members of affiliated volunteer responder groups (e.g., members of 

Community Emergency Response Teams and Medical Reserve Corps)

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness
28

a. Applicability across communities of varying size and type – The approach is applicable 

across communities of varying size and type.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc. – CART is not hazard-specific. Thus, it is broadly applicable.

c. Applicability across different recovery time scales – Unclear from the available materials.

d. Representativeness across different systems/interdependencies – CART is not presented 

in a way that focuses on interdependencies. It targets the following overlapping, 

interrelated domains that both describe and affect community resilience:

 Connection and Caring – Includes participation, relatedness, shared values, 

support systems, a sense of fairness, and hope. 

 Resources – Includes natural, physical, human, financial, and social resources. 

 Transformative Potential – Includes the ability to identify and frame collective 

experiences, data collection and analysis, planning, and skill building that create 

the potential for community change. 

 Disaster Management – Includes disaster prevention and mitigation, 

preparedness, response, and recovery.

2. Utility

a. User-friendliness – Very user-friendly.

b. Level of technical resources required to implement the methodology – The necessary 

tools are provided. It is quite labor and time intensive, but necessarily so. The more 

robust the local knowledge/capacity/infrastructure, the more sophisticated the result. 

Would be best if some facilitation and/or social science expertise were involved in the 

implementation.

c. Value of methodology outputs in supporting resilience planning – High value here. 

Begins by identifying current assets and moving forward with strength, weakness, 

opportunity, and threat (SWOT) analysis. The focus of this approach is process; it is 

intended to engage communities in thinking about resilience and provide a foundation to 

move forward into more sophisticated activities.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? – 

Yes. 

“Resilience can be thought of as an attribute (an ability or capacity), a process, and/or an 

outcome associated with successful adaptation to, and recovery from, adversity. Building 

a resilient community involves more than assembling a collection of resilient individuals. 

Community resilience requires that the community as a whole must cope effectively with 

and learn from adversity. A resilient community has the ability to transform the 

environment through deliberate, collective action.”

3. Ability to assess/measure/predict...

a. Physical impacts, interdependencies, and recovery times –

b. Economic impacts and recovery times –

c. Social impacts and recovery times –

d. Ecological impacts and recovery times
29

The data collection framework is quite comprehensive (see Appendix C of the CART report). It 

covers each of the essential dimensions of community resilience, but the extent to which and how 

this is approached by individual communities will vary. 

Tools included in the assessment process include templates or instruments for each of the 

following:

 Communities Advancing Resilience Toolkit (CART): The CART Integrated System© 

o CART Assessment Survey tool

o Key Informant Interviews tool

o Data Collection Framework tool (see below)

o Community Conversations tool

o Neighborhood Infrastructure Maps tool

o Community Ecological Maps (Eco-Maps) tool

o Stakeholder Analysis tool

o SWOT Analysis tool

o Capacity and Vulnerability Assessment tool

 Appendices

o Survey Instruments 

o Sample Questions for Key Informant Interviews and Community Conversations 

o Templates

Specifically, the data collection framework tool includes the following (see section 4, below, for 

a description of each of the tools):

 Demographic data 

o Sex 

o Age 

o Race and ethnicity 

o Disabilities 

o Employment and unemployment 

o Income and poverty 

o Health status 

o Education attainment 

o Literacy 

o Primary language 

o Marital status 

o Religious affiliations 

o Other

 Household data 

o Children per household 

o Elders per household 

o Vehicles available/operational 

o Phones by type (landline, cell) 

o Heating fuel by type (e.g., electricity, coal, wood) 

o Cooling system by type (e.g., air conditioner, evaporative cooling, fan) 

o Other 

 Housing data 

o Occupancy 

o Home ownership 
30

o Multi-unit developments 

o Homeowner associations 

o Length of home ownership 

o Other

 Education 

o Schools 

o Colleges and universities 

o Vocational/technical schools 

o Apprentice and internship opportunities

o High school equivalency and adult literacy programs 

o Job training organizations (e.g., Job Corps, Goodwill job training) 

o Other 

 Business, occupational, and economic data 

o Major and minor industries by type 

o Major and minor businesses by type 

o Occupations by type 

o Licensing boards 

o Labor unions 

o Employment agencies 

o Employment/career centers 

o Office parks 

o Shopping malls 

o Hotels and motels 

o Restaurants 

o Professional associations 

o Other 

 Transportation data 

o Fuel stations 

o Bus stations 

o Train stations 

o Elevated trains, light rail stations, subways 

o Airports 

o Highways 

o Bridges 

o Tunnels 

o Other 

 Health and human services 

o Hospitals 

o Clinics 

o Nursing homes 

o Assisted living facilities 

o Social service providers by type 

o Physicians 

o Mental health service providers

o Support groups 

o Food banks 

o Existing shelters 

o Contingent shelters 

o Sanitation services 

o Other 
31

 Faith-based organizations 

o Churches, synagogues, mosques 

o Other 

 Libraries

 Recreation opportunities 

o Parks 

o Sports groups 

o Sports facilities (e.g., bowling alleys, ice rinks) 

o Other 

 Civic and Fraternal Organizations

 Media 

o Newspapers 

o Radio stations 

o Television stations 

o Other 

 Internet Penetration

 Public Meetings 

o Public forums 

o Governing council meetings 

o Other 

 Voting 

o Availability of voter registration sites 

o Availability of polling sites 

o Registered voters 

o Voter turnout

o Other

 Organized volunteerism 

o Number and type of organizations 

o Participation in organized volunteer activities 

o Other 

 Philanthropy 

o Number and type of organizations 

o Contributions 

o Other 

 Crime data 

 Other

4. Which of the following techniques are used in the methodology? This will vary by community.

a. Checklists - Yes

b. Interviews - Yes

c. Ratings - Yes

d. Physical inspections - No

e. Exercises (e.g., tabletop) - No

f. Engineering analyses - No

g. Statistical inference - No

h. Simulations - No
32

i. Surveys – Yes

The CART tools are described by the authors as follows (Reference 1):

CART Assessment Survey 

This field-tested survey instrument assesses a community’s resilience across the four CART 

domains, explores participants’ personal relationship to their community, and queries standard 

demographics. The CART survey design encourages the addition of items to address specific 

concerns for your organization and/or community. 

Key Informant Interviews 

Key informant interviews generate qualitative information from individuals who are 

knowledgeable about your community. Instructions provide guidance for selecting key informants 

and conducting interviews. Sample questions are included to address various aspects of 

community resilience, each of the four CART domains, terrorism preparedness, and public 

engagement. 

Data Collection Framework 

The data collection framework identifies the type and sources of existing data that may be useful 

in a community’s strategic planning to build community resilience to terrorism and disasters. A 

listing of the specific data fields in the data collection framework tool is provided above under 

Question 3.

Community Conversations 

Community conversations foster the exchange of information, ideas, and opinions among 

members of your community. Instructions help facilitators to convene and conduct conversations. 

Sample questions are provided. 

Neighborhood Infrastructure Maps 

This tool provides guidance for mapping the physical infrastructure of a neighborhood. Such 

maps improve your awareness of a neighborhood. They can be used to introduce your 

organization to a community and to initiate networking with neighbors. 

Community Ecological Maps 

This visual too for describing the nature and strength of relationships within a community can 

provide a foundation for improving relationships and partnerships. 

Stakeholder Analysis 

Stakeholder analysis will help you to analyze the potential influence of key individuals, groups, 

and organizations. It provides information to garner support for, and limit opposition to, your 

activities. 

SWOT Analysis 

This strategic planning tool provides information for analyzing a community’s resources and 

capabilities. Additionally, the tool aids in developing strategies to utilize strengths, pursue 

appropriate opportunities, overcome weaknesses, and reduce vulnerabilities. 

Capacity and Vulnerability Assessment 

This tool is particularly useful in communities where groups may be impacted differently 

depending on, for example, socio-economic status, health, or ethnicity.
33

5. Critical Assessment

a. Maturity – Report published in 2012 (Reference 1). Article about one implementation in 

2014 (Reference 2).

b. Uniqueness/innovativeness – CART is innovative in that it provides a complete set of 

tools and guidelines for communities to engage in an assessment of their resilience across 

a number of domains—to encourage a process for moving forward to gather specific 

information and develop a robust plan.

c. Objective and repeatable? – The methodology is replicable. The way communities will 

go about their detailed assessment activities is not standardized; this system relies on 

local subject matter expertise, which will likely produce differing assessment approaches

(in terms of what gets input into the system) and thus non-standardized results.

d. Scientific basis/merit/empirical evidence to support the method

i. Current/prior use of methodology – The authors report that there have been some 

early survey applications, including a sample of Community Emergency 

Response Team (CERT) members in Arizona, two community samples, and a 

university-wide survey sample. The authors are preparing manuscripts describing 

the early community samples and plan to report on these studies in the near 

future.

ii. Extent to which the approach been field tested – The authors indicated that the 

assessment survey has been field tested. One article was found that describes 

CART’s implementation with a convenience sample of 64 study participants who 

were members of affiliated volunteer responder groups (e.g., members of 

Community Emergency Response Teams and Medical Reserve Corps). See 

Reference 2.

iii. Extent to which the approach has been self-assessed – Unable to locate any 

evaluative information.

iv. Has it been applied to real-world planning and management? – Unable to locate 

anything publicly available.

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? – Unable to locate any evaluative information.

vi. Clarity – The process itself is very clear; what is not entirely clear is whether the 

complete toolkit has been implemented anywhere

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? – N/A

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? The approach would 

need to focus more on systems and interdependencies. The process-related 

aspects of CART could be adapted/included in other approaches that have a more 

robust methodology.

ii. What are the next steps that must be taken to improve the methodologies 

currently available? More field-testing would be advised.
34

This Page Intentionally Left Blank
35

8. Assessment of Baseline Resilience Indicators for Communities 

(BRIC)

References:

1. 2014. S Cutter, K Ash, and C Emrich. See “The geographies of community disaster resilience,” 

retrieved 12/09/14 at http://www.sciencedirect.com/science/article/pii/S0959378014001459

Summary:

This process builds on prior work by Cutter et al. and is based on empirical research with solid conceptual 

and theoretical underpinnings. The purpose of BRIC is to measure overall pre-existing community 

resilience. Using U.S. Counties as the study unit, the approach provides an empirically-based resilience 

metric for use in a policy context. It uses data from public and freely available sources, drawing on a 

common set of 49 indicators associated with six different domains: 

 Social (10 indicators)

o Educational attainment 

o % of population below 65 years of age

o % of households with at least one vehicle

o % of households with telephone service available

o % of population that are proficient English speakers

o % of population without disabilities

o % of population under age 65 with health insurance

o Mental health support (psychosocial support facilities per 10,000 persons)

o Food security rate

o Physicians per 10,000 persons

 Economic (8 indicators)

o % of owner-occupied housing units

o Employment rate

o Race/ethnicity income equality (negative Gini coefficient)

o % of employees not in farming, fishing, forestry, extractive industry, or tourism

o Gender income equality

o Ratio of large to small businesses

o Large retail stores per 10,000 persons

o Federal employment

 Housing and infrastructure (9 indicators)

o % of housing units that are not manufactured homes

o % of vacant units that are for rent

o Hospital beds per 10,000 persons
36

o Evacuation routes (major road egress points per 10,000 persons)

o % of housing units built prior to 1970 or after 2000

o Hotels/motels per 10,000 persons

o Public schools per 10,000 persons

o Rail miles per square mile

o % of population with access to broadband internet service

 Institutional (10 indicators)

o Average per capita spending for mitigation projects

o % of housing units covered by National Flood Insurance Program

o Governments and special districts per 10,000 persons

o Presidential disaster declarations per loss-causing hazard event (2000-2009)

o % of population in communities with Citizens Corps program

o Proximity of county seat to state capital

o Proximity of county seat to the nearest Metropolitan Statistical Area

o Population change over previous five year period

o Nuclear accident planning (% of population within 10 miles of a nuclear power plant)

o Crop insurance policies per square mile

 Community Capital (7 indicators)

o % of population not foreign-born persons who came to U.S. within previous five years

o % of population born in state of current residence

o % of voting age population participating in presidential election

o Persons affiliated with a religious organization per 10,000 persons

o Civic organizations per 10,000 persons

o Red cross volunteers per 10,000 persons

o Red cross training workshop participants per 10,000 persons

 Environmental (5 indicators)

o Farms marketing products through Community Supported Agriculture per 10,000 persons

o Natural flood buffers (% of land in wetlands)

o Efficient energy use (megawatt hours per energy consumer)

o Pervious surfaces (average percent perviousness)

o Efficient water use (Inverted water supply stress index)

The BRIC is broken down into the aforementioned domains, so that policy makers can determine areas in 

which to invest in intervention strategies to improve resilience scores.

Data sources and time periods used to construct resilience indicator set:

 United States Federal Government:
37

1. USA Counties Database, 2007, Census Bureau

2. County and City Data Book, 2007

3. County Business Patterns, 2009–2010

4. Decennial Census, 2010

5. Small Area Health Insurance Estimates, 2010

6. Tiger/Line, 2010

7. Current Population Estimate, 2005, 2012

8. American Community Survey Three-Year Estimates, 2008–2010

9. American Community Survey Five-Year Estimates, 2006–2010

10. Hazard Mitigation Grant Program, 2000–2009, Federal Emergency Management Agency

11. Presidential Disaster Declarations Database, 2000–2009

12. Citizen Corps Councils, 2010

13. National Flood Insurance Program, 2010

14. National Land Cover Dataset, 2006, U.S. Geological Survey

15. National Atlas, 2010

16. Quarterly Census of Employment and Wages, 2010, Bureau of Labor Statistics

17. Census of Agriculture, 2007, Department of Agriculture

18. National Center for Education Statistics, 2009–2010, Department of Education

19. Electricity Consumption, 2010, Energy Information Administration

20. Broadband Internet Access, 2010, Federal Communications Commission

21. Water Supply Stress Index, 2005, Forest Service

22. Nuclear Power Plants Database, 2010, Nuclear Regulatory Commission

23. Railroad Network, 2010, Oak Ridge National Laboratory

 Academic:

24. Spatial Hazard Events and Losses Database for the US (SHELDUS), 2000–2009, Univ. 

South Carolina Hazards and Vulnerability Research Institute

 Nonprofit/Open Access:

25. Religious Congregations and Membership Study, 2010, Association of Religion Data 

Archives

26. Farm Subsidies, 2010, Environmental Working Group

27. Map the Meal Gap, 2010, Feeding America

28. US 2012 Presidential Election, 2012, The Guardian

29. Volunteers and Preparedness Training, 2013, American Red Cross

 Proprietary:

30. Million Dollar Database, 2010, Dun and Bradstreet
38

Additional notes:

As presented, there is little community engagement in the assessment process; this does not seem to 

preclude presenting the findings of the BRIC assessment to local stakeholders to involve them in 

resilience planning processes and prioritization of local needs.

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness

a. Applicability across communities of varying size and type – this is a county-level 

approach.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc. – applicable across different types of hazards

c. Applicability across different recovery time scales – New data would need to be collected 

to address this (i.e., this approach focuses on baseline)

d. Representativeness across different systems/interdependencies – does not seem to address 

system interdependencies as we are approaching them in the context of the NIST work; 

however, the model may take this into account

2. Utility

a. User-friendliness – it is user-friendly, but does require expertise in the analysis of data

b. Level of technical resources required to implement the methodology – requires data 

analysis expertise

c. Value of methodology outputs in supporting resilience planning – high; the methodology 

itself focuses on providing information to use with respect to planning

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? –

Yes

3. Ability to assess/measure/predict...

a. Physical impacts, interdependencies, and recovery times

b. Economic impacts and recovery times

c. Social impacts and recovery times

d. Ecological impacts and recovery times

For each of the above, BRIC assesses baseline information for these various potential impacts of 

a disaster using existing data; it does not examine interdependencies or recovery times.

4. Which of the following techniques are used in the methodology?

a. Checklists - No

b. Interviews - No

c. Ratings - No

d. Physical inspections - No

e. Exercises (e.g., tabletop) - No

f. Engineering analyses - No

g. Statistical inference - Yes
39

h. Simulations - No

5. Critical Assessment

a. Maturity – Relatively new methodology, but builds on the Social Vulnerability Index.

1

b. Uniqueness/innovativeness – Innovative in that it focuses on characteristics that seem to 

have the greatest impact on resilience

c. Objective and repeatable? Yes

d. Scientific basis/merit/empirical evidence to support the method

i. Current/prior use of methodology – Cutter’s team has applied the methodology to 

U.S. Counties and published the assessment in peer-reviewed journals

ii. Extent to which the approach been field tested - same as above

iii. Extent to which the approach has been self-assessed – it has been peer-reviewed 

and has traction in the social science community as an approach to assess 

baseline (for which it is intended).

iv. Has it been applied to real-world planning and management? – Not clear

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? – Nothing published on this

vi. Clarity- The methodology is clearly defined

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? – The 49 indicators 

were selected through conceptual, theoretical, and/or empirical justification as 

capturing qualities associated with community resilience. The final composite 

scores are computed using two different methods: (1) “a deductive binning of 

indicators into resilience categories and subsequent summations within and 

across the categories” and (2) “an inductive indicator method constructed via

principal components analysis and subsequent summation of factor scores across 

the first six components.” The BRIC model developers state in Reference 1 that 

the two methods produced similar results.

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? This approach is 

intended to provide baseline information for resilience planning. The data it 

provides would likely be useful as part of a broader process and would not be a 

standalone substitute for more comprehensive, community-based efforts.

ii. What are the next steps that must be taken to improve the methodologies 

currently available? As with other resilience metrics, the BRIC model developers 

acknowledge in Reference 1 that the methodology does not “address the 

persistent difficulty in validating whether quantitative indicator models of social 

vulnerability or disaster resilience truly represent such abstract concepts. This 

remains an important open question.”

 

1 Cutter, S. L., B. J. Boruff and W. L. Shirley (2003). "Social vulnerability to environmental hazards," Social 

Science Quarterly 84(1): 242-261.
40

This Page Intentionally Left Blank
41

9. Assessment of Rockefeller Foundation City Resilience Framework

(CRF) and City Resilience Index (CRI)

References:

1. http://www.rockefellerfoundation.org/uploads/files/0bb537c0-d872-467f-9470-b20f57c32488.pdf

2. http://www.rockefellerfoundation.org/blog/framework-articulating-city-resilience

Summary (excerpts from Reference 1):

This report presents ... [a] framework for articulating city resilience ... to underpin the City Resilience 

Index. It has already proven useful in the agenda-setting workshops in cities across the globe that are 

participating in the 100 Resilient Cities Challenge. These workshops, in turn, have helped and will 

continue to help shape the framework and contribute to the final phase, developing the indicators and 

variables that will comprise the City Resilience Index.

This framework will form the basis of a tool that should enable all of us interested in city resilience to 

convene around a common understanding of that idea, and begin to ‘baseline’ what matters most for 

making cities more resilient. Both the framework and the index are intended to facilitate a process of 

engagement with and within cities that generates dialogue and deeper understanding. Ultimately, this will 

lead to new ideas and opportunities to engage new actors in civil society, government and business on 

what makes a city resilient.

The City Resilience Framework provides a lens through which the complexity of cities and the numerous 

factors that contribute to a city’s resilience can be understood. It comprises 12 key indicators that 

describe the fundamental attributes of a resilient city.

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness

a. Applicability across communities of varying size and type

Generally, yes, although the focus is mostly on cities and less on rural communities.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, ...

Yes, substantial. Although the initial motivation for the framework is climate change, the 

methodology can be applied to a variety of events and circumstances.

c. Applicability across different recovery time scales

Yes, although the intention of the framework is to be a tool for city planners and, thus, 

tends to provide a longer time horizon than frameworks that just focus on disaster 

response.

d. Representativeness across different systems/interdependencies

Yes. The Rockefeller Foundation City Resilience Framework (CRF) is very 

comprehensive in considering different systems, both physical and social. 

Interdependencies are implicitly acknowledged for both social and physical systems, for 

example, by indicating that “Health services encompass a variety of practices, including: 

education; sanitation; epidemiological surveillance; vaccination; and provision of 

healthcare services.”
42

2. Utility

a. User-friendliness. Somewhat. Still the City Resilience Index (CRI) is under development 

so the details of how simple its application will be are not completely known.

b. Level of technical resources required to implement the methodology. Average.

c. Value of methodology outputs in supporting resilience planning. Very high. One of the 

main goals of the CRF is to be applied in the planning process.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? 

Somewhat. Most of the common aspects are in the long term aspects of PPD-21 

definition—the preparation and adaptation portions—and less in the shorter term aspects 

related to natural disasters—withstanding and recovery speed.

3. Ability to assess/measure/predict... 

a. Physical impacts, interdependencies, and recovery times.

b. Economic impacts and recovery times

c. Social impacts and recovery times

d. Ecological impacts and recovery times

The primary areas of focus of the CRF and CRI appear to be economic aspects and social and 

ecological impacts.

4. Which of the following techniques are used in the methodology?

a. Checklists - Yes

b. Interviews - Yes

c. Ratings - Yes

d. Physical inspections

e. Exercises (e.g., tabletop)

f. Engineering analyses

g. Statistical inference

h. Simulations

i. Workshop - Yes

5. Critical assessment

a. Maturity: About halfway through its development. A document describing the CRF has 

been issued for comments. The most significant development need is on the CRI. 

Although it was previously indicated that this index would be presented on December 

2014, as of November 2015 the CRI does not appear to have been publicly released. 

Once it is released, the next important step towards completing development would be to 

validate those metrics in a practical context.

b. Uniqueness/innovativeness Somewhat.

c. Objective and repeatable? Not clear. The index is still under development although 

existing reports seem to indicate that the CRI will be repeatable and with substantial 

scientific background supporting a mostly objective method.
43

d. Scientific basis/merit/empirical evidence to support the method. Yes, work has been 

conducted on a scientific basis, including a complete literature review and field work to 

collect critical data and analyze it.

i. Current/prior use of methodology. Since it is still under development, use of this 

methodology cannot be assessed, yet. However, the Rockefeller Framework is 

also being used in their Resilience Academies that are being held in support of 

the U.S. Department of Housing and Urban Development’s National Disaster 

Resilience Competition.

ii. Extent to which the approach been field tested. It does not seem to have been 

tested, yet.

iii. Extent to which the approach has been self-assessed. It does not seem to have 

been self-assessed, yet.

iv. Has it been applied to real-world planning and management? No, with the 

exception of a few test cases. 

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? Yes, but still within the context of cities around the world.

vi. Clarity – Very clear.

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? The metric under 

development is attempting to be able to aggregate components into a single 

metric. Still, it is unclear how this will be done.

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? 

Social and physical metrics need to be better integrated and objective metrics for 

each individual physical or social system need to be identified and combined 

within a single community framework. Such metrics need to take into account 

the interdependent relationship not only among physical infrastructure systems

but also among physical infrastructure and social systems. The research should 

also facilitate the goal of reaching a single community resilience metric without 

the risk of using somewhat subjective weight factors or relying on complex 

questionnaires processes. Eventually, the frameworks need to be validated with 

field data.

ii. What are the next steps that must be taken to improve the methodologies 

currently available? 

Significant efforts in data collection will need to be undertaken in order to 

provide the necessary inputs to the framework to validate new or existing 

frameworks. In parallel, substantial interdisciplinary research combining social 

and technical fields with data analytics will be necessary in order to achieve the 

goal of a fully integrated, yet simple, resilience framework.

Evaluation

Despite the fact that the CRF is the initial step towards the development of a CRI, the framework shows a 

very comprehensive development process with significant attention towards reaching an objective and 
44

simple metric for community resilience. One of the merits of this framework is that it is based on what 

seems to be a well thought out research process with a very extensive literature review on a broad set of 

environmental conditions (i.e., cities with different characteristics) and with substantial field work to 

collect data and develop case studies. Sandia National Lab is actively supporting the 100 Resilient Cities 

Program. This framework seems to balance both depth and breadth in how resilience is assessed. 

Objectivity and complexity seem to have been well considered in a balanced way.

As the basis for developing a CRI (which was expected to be completed by the end of 2014), the CRF 

explains the research methodology and process to achieve the goal of measuring resilience in an objective 

quantitative way by defining its structure and its constituent categories, the indicators and sub-indicators, 

as well as a preliminary list of variables and metrics. As the two figures reproduced below (Reference 1)

represent, the framework organizes 12 so-called key indicators in the following 4 categories:

 Leadership and strategy

 Health and wellbeing

 Infrastructure and environment

 Economy and social

This organization seems to integrate social and infrastructure aspects well, and to consider human-driven 

processes inherent in the system-of-systems making up the community fabric of a city. 

Economic/financial constraints are also considered in an integral way providing a realistic setting for its 

application for planning purposes. The 12 indicators used to assess performance are:

1. Minimal human vulnerability

2. Diverse livelihoods and employment

3. Adequate safeguards to humans life and health

4. Collective identity and mutual support

5. Social stability and security

6. Availability of financial resources and contingency funds

7. Reduced physical exposure and vulnerability

8. Continuity of critical services

9. Reliable communications and mobility

10. Effective leadership and management

11. Empowered stakeholders

12. Integrated development and planning

In turn, these 12 key indicators span 7 attributes of what is considered a resilient city: being reflective, 

resourceful, robust, inclusive, redundant, integrated, and/or flexible.
45

Figure 1. General Structure of the City Resilience Framework (CRF) 

(http://www.rockefellerfoundation.org/uploads/files/0bb537c0-d872-467f-9470-b20f57c32488.pdf)

Figure 2. Detailed CRF Structure

(http://www.rockefellerfoundation.org/uploads/files/0bb537c0-d872-467f-9470-b20f57c32488.pdf)
46

This Page Intentionally Left Blank
47

10. Assessment of NOAA Coastal Resilience Index

References:

1. http://masgc.org/assets/uploads/publications/662/coastal_community_resilience_index.pdf 

2. http://masgc.org/coastal-storms-program/resilience-index 

3. https://www.fema.gov/media-library/assets/documents/4241

4. Thompson, J., T. Sempier, and L. Swann. “Evaluation of the Community Resilience Index – A 

Community Planning Tool.” Auburn University Marine Extension and Research Center and 

Mississippi-Alabama Sea Grant Consortium. September 2014.

Summary:

The National Oceanic and Atmospheric Administration’s (NOAA) Coastal Resilience Index was 

developed in November 2010 with a purpose of providing a simple and inexpensive self-assessment tool 

to give community leaders a method of predicting if their community will reach and maintain an 

acceptable level of functioning after a disaster. The tool was meant to be completed by experienced local 

planners, engineers, floodplain managers and administrators in less than three hours using readily 

available, existing sources of information, using a yes/no question format. 

The eight page assessment form addresses six broad areas: 

1. Critical facilities and infrastructure

2. Transportation issues

3. Community plans and agreements

4. Mitigation measures

5. Business plans

6. Social systems

The resulting assessment is meant to identify problems (vulnerabilities) that should be addressed before 

the next disaster; areas in which a community should become more resilient; and where resources should 

be allocated. It also estimates the adaptability of a community to a disaster, but is not meant to replace a 

detailed study. The authors note that “The Resilience Index and methodology does not replace a detailed 

study.... But, the Resilience Index resulting from this Community Self-Assessment may encourage your 

community to seek further consultation” (p. 1). 

The authors also state that the tool should not be used to compare one community to another; rather, they 

recommend it be employed as an approach for internal evaluation to identify areas in which a given 

community might increase its resilience. The description suggests the FEMA Guide to “Understanding 

Your Risks: Identifying Hazards and Estimating Losses” (Reference 3) would be used to do more detailed 

and quantitative comparative studies.

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness

a. Applicability across communities of varying size and type

This assessment method could be applicable across different sized areas. However, based 

on the type of questions and the types of hazards that are discussed, it appears to be 

targeted more at small-to-medium-sized coastal communities/cities. More specifically, 

the NOAA resilience index procedures were stated to be targeted at coastal communities 
48

in the five U.S. Gulf states that include Texas, Louisiana, Mississippi, Alabama, and 

Florida.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc.

Based on the type of questions posed in the assessment as well as the scenario inputs into 

the tool, this method is targeted primarily at coastal storms, particularly hurricanes and 

other surge or rain induced flooding events.

c. Applicability across different recovery time scales 

The primary focus of this assessment tool is on immediate and short term recovery, more 

specifically the “restoration of basic services” and how long a community will take to 

reach and maintain an acceptable level of functioning after a disaster. This includes 

reoccupying homes and businesses. The time scales discussed in the results interpretation 

include: “less than two months” for a MEDIUM Resilience Index; “less than two weeks” 

for a HIGH Resilience Index; and “more than 18 months” for a LOW Resilience Index.

d. Representativeness across different systems/interdependencies

The focus of the index rating is on six main areas including: (1) critical facilities and 

infrastructure; (2) transportation issues; (3) community plans and agreements; (4) 

mitigation measures; (5) business plans; and (6) social systems. Although some 

discussion is made of general interdependencies (for example, the reopening of retail 

stores and the availability of utilities to support them), there is little, if any, detailed 

discussion of the topic or the concept.

2. Utility

a. User-friendliness

The Coastal Resilience Index approach uses a simple series of check boxes that can be 

filled out by the reviewer. At the end of the assessment, the number of checks is totaled 

and a resilience index (LOW, MEDIUM, HIGH) is given along with an estimated 

“percentage of critical facilities functioning after a disaster.” 

b. Level of technical resources required to implement the methodology 

Although it is not explicitly stated as a requirement, the NOAA CRI would require 

considerable expertise on the part of the evaluator, specifically knowledge of the location 

and vulnerability of critical transportation, utility infrastructure, and other critical 

community facilities (police stations, hospitals, shelters, etc.). It also requires knowledge 

of community plans, personnel or cooperative agreements; building codes; business 

continuity plans; and community civic/cultural plans. The index instructions suggest that 

these are all expected to be “readily and easily available,” but this cannot always be

expected to be the case.

c. Value of methodology outputs in supporting resilience planning 

Similar to many other evaluation methods that were reviewed in the transportation sector, 

the NOAA Coastal Resilience Index computation method appears to focus primarily on 

the identification of sector vulnerabilities with much less attention paid to the methods 

and ideas for recovery. It is also interesting to note that while the Index ratings of LOW, 

MEDIUM and HIGH correlate to the amount of time it is assumed to take for “basic 

services are restored,” it is not readily apparent how these recovery durations were 

estimated. It is assumed that there is some type of correlation with vulnerability, 

whereby highly vulnerable facilities and infrastructure would be expected to sustain 
49

significant damage and therefore would be likely to take correspondingly longer to 

recover.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21?

In general, it appears that this method has consistencies with the Directive. However, it is 

notable that there are differences in specific definition language. For example, there are 

differences in the specific definitions for both of the terms included below.

i. Resilience = “The ability to prepare for and adapt to changing conditions and 

withstand and recover rapidly from disruptions.”

“DISASTER RESILIENCE is the capacity of a community exposed to hazards to 

adapt, by resisting or changing, in order to reach and maintain an acceptable level 

of functioning and structure.”

ii. Includes the ability to withstand and recover from deliberate attacks, accidents, 

or naturally occurring threats or incidents

“RESILIENCE is determined by the degree to which the community is capable 

of organizing itself to increase its capacity for learning from past disasters.”

3. Ability to assess/measure/predict...

a. Physical impacts, interdependencies, and recovery times

The NOAA Index includes each of the above sector areas. It also goes into more detail 

and specifics on each, targeting in particular key elements and facilities associated with 

evacuation, sheltering, and health care. In terms of community-level planning there is 

emphasis on flooding and natural disasters (like hurricanes). While there is also emphasis 

on mitigation efforts, business planning, and social systems, the evaluation methodology 

does not specifically discuss them within the context of interdependency need or 

importance. As discussed previously, recovery time is also a component of the evaluation 

process, however, it is not clear what methods or measures were used to determine how 

these times were estimated.

b. Economic impacts and recovery times

The methodology does not explicitly discuss or model economic impacts, but there is 

discussion of business planning, primarily as it pertains to business continuity in the 

aftermath of a natural disaster. The list below is included as a suggestive checklist of 

major issues relative to continuing business activity and the ability of retail operations to 

support the recovery of communities impacted by a disaster.

 Generators

 Backup options for basic needs (water, sewer, food, and communications)

 Plans to bring in staff to help reopen the business (considering impacts to staff)

 Plans for restocking

 Plans for ice distribution

c. Social impacts and recovery times

Similar to the areas discussed previously, the NOAA Index methodology does not 

explicitly seek to model social impacts in detail. Once again, the discussion of social 

impacts is geared at social networks and support systems as they may pertain to 

natural disaster preparedness, response, and recovery. The list below is included as a 
50

suggestive checklist of issues included in the assessment guide that are related to 

these needs:

 Strong faith-based networks (counted on during a disaster)

 Cultural identity (unified Hispanic, Asian or other ethnic communities)

 Neighborhood associations - Support members in times of need

 Business cooperative or working relations (industries that employ many 

residents, Chamber of Commerce, other Business-related networks, etc.)

 Strong civic organizations (Kiwanis Club, Rotary Club, etc.)

a. Ecological impacts and recovery times – No

4. Which of the following techniques are used in the methodology?

a. Checklists - The assessment foundation is based on six yes/no checklists that incorporate 

the six assessment areas. However, given the vague/broad nature of the checklists, it 

would be assumed that considerable expertise would be vital for the person completing 

the checklist and, while not explicitly stated in the methodology, the completion of the 

checklists would significantly benefit from, if not require, all of the measures listed 

below.

b. Interviews

c. Ratings

d. Physical inspections

e. Exercises (e.g., tabletop)

f. Engineering analyses 

g. Statistical inference

h. Simulations

5. Critical Assessment

a. Maturity

This assessment method guide was published in 2010. As part of its development process 

the NOAA Community Resilience Index (CRI) was pilot tested in 17 communities in five 

states (Alabama, Florida, Louisiana, Mississippi, and Texas). In addition to developing 

their community indices, these pilot tests were also used to further refine and improve the 

assessment methodology.

After the pilot tests were conducted, updates and revisions were made to the report, and 

two follow-up recommendations were made:

“Follow up with each community and collect data which denotes how the 

community has used the index and what changes they have made as a result of 

completing the index. Also collect quantitative data that compares the original 

score a community received on the index to their score after a 6-month or 1-year 

period has occurred to see if there are any changes.”

Although developed specifically for the Gulf Coast region of the US, the method is also 

now being suggested (which appropriate modification), in the New England and Pacific 

regions of the U.S. as well as for use in Mexico and Bangladesh. However, no specifics 
51

on what modifications would be needed or how the method would vary from the Gulf 

Coast version were included.

b. Uniqueness/innovativeness

There was nothing particularly unique or innovative in this evaluation methodology. 

Most of the topics were similar to those discussed elsewhere. It could be suggested that 

this method was unique in that it emphasize simplicity, low cost, and time or effort. One 

area that was interesting was the emphasis in the Implementation and Evaluation sheet on 

training; collaboration between agencies and officials in states in which it was planned 

for use; and making revisions/updates to the system were notable concepts which were 

not discussed in other evaluation methods that were reviewed in this effort. This is quite 

similar to other programs in the hurricane preparedness/response/recovery area in which 

it is fairly common to see federally-led training programs (most notably, FEMA) that 

seek to inform state and local emergency management officials and seek to establish 

person-to-person collaborative relationships within and between participating agencies, 

jurisdictions, and states.

c. Objective and repeatable? 

The methodologies and techniques included in the NOAA Resilience Index process were 

repeatable in terms of the application of the methods themselves. However, the checklists 

are quite broad and subjective as to whether a particular reviewer feels that the specific 

criteria warrant a check or not. An exception to this is the physical location of critical 

infrastructure and facilities with respect to an established flood area. These locations 

would come directly from maps and would be quite objective and repeatable between two 

or more separate reviewers.

d. Scientific basis/merit/empirical evidence to support the method 

The NOAA CRI relies heavily on the experience and expertise of the local team 

conducting the evaluation. Specifically, the method seeks input from “experienced local 

planners, engineers, floodplain managers, or administrators” who complete the survey 

forms using “existing sources of information from their community.” Thus, while there 

is an implied scientific basis to the method, it is not quantitative, nor particularly 

systematic and could, therefore, could yield highly variable results based on the 

perceptions of the team completing the survey

e. Current/prior use of methodology

i. Extent to which the approach been field tested

The self-assessment process was pilot tested in 17 communities in Texas, 

Louisiana, Mississippi, Alabama, and Florida.

ii. Extent to which the approach has been self-assessed

The NOAA CRI was self-assessed and the results documented as part of a 2014 

report cited above. The self-assessment report was documented to find areas 

where the method could be improved and made more comprehensive and userfriendly. The assessment also revealed techniques on how to better conduct the 

assessment surveys. The self-assessment also included results (and helpful 

comments) from a user-survey

iii. Has it been applied to real-world planning and management?
52

As stated above, the NOAA CRI process was pilot tested in 17 communities in 

Texas, Louisiana, Mississippi, Alabama, and Florida. These results were used to 

support actual resilience planning efforts in all of these locations.

iv. Does evaluative information exist regarding how the methodology has worked in 

different settings?

The evaluation was focused more on the conduct of the assessment rather than 

the results. However, all of the evaluation teams reported that the method was 

indeed useful for helping them to identify areas for improving resilience and 

assessing vulnerabilities in their local communities

v. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined?

The outcomes of the five individual, sector-specific, steps of the method are 

combined to give an overall “score” of community resilience (low/medium/high). 

This overall rating is then used to broadly translate this ranking into an expected 

time duration for which basic community services would be restored to 

acceptable levels of service

f. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach?

ii. What are the next steps that must be taken to improve the methodologies 

currently available?

Although the self-assessment report did not mention how the assessment survey 

results would be used to specifically address such needs and gaps, it did identify 

several specific areas of potential recommended improvement. These included 

the needs to:

 Update the assessment on an annual basis. However, caution was also 

suggested to limit participant “fatigue” from participating in too many such 

efforts and exercises.

 Allow adequate time to lapse between meetings for “significant changes to 

occur” since the development and implementation of these resilience 

improvement concepts can take up to several years. Frequent reexaminations of the local CRI “may not give enough time for previous 

results to demonstrate an effect.”

 Try to maintain consistency of evaluators between meetings. Although this 

may often be difficult due to personnel turn-over, it would likely help to 

increase “institutional memory” during the longitudinal implementation 

process.

 Assuring that the appropriate community representatives complete the 

assessments. While it is often required to have an assistant attend the 

meeting, the nature of the NOAA CRI method relies heavily on the input of 

experienced experts.

Other areas of need noted by the participants included the need for more 

information on “business plans, social systems, and community plans and 
53

agreements.” The specifics of what should be included in these areas were not 

necessarily made clear in the assessment description. Similar views were echoed 

relative to issues of “communication across socio-economic sectors, noting that 

vulnerable minority populations were not included in the exercises.” The 

participants felt that each of these concepts as well as transportation issues were 

not detailed adequately in the assessment.
54

This Page Intentionally Left Blank
55

11. Assessment of FEMA Hazus Methodology

References:

1. https://msc.fema.gov/portal/resources/hazus

Summary (Reference 1):

“Hazus is a nationally applicable standardized methodology that contains models for estimating potential 

losses from earthquakes, floods and hurricanes. Hazus uses Geographic Information Systems (GIS) 

technology to estimate physical, economic and social impacts of disasters. It graphically illustrates the 

limits of identified high-risk locations due to earthquake, hurricane and floods. Users can then visualize 

the spatial relationships between populations and other more permanently fixed geographic assets or 

resources for the specific hazard being modeled, a crucial function in the pre-disaster planning process.

Hazus is used for mitigation and recovery, as well as preparedness and response. Government planners, 

GIS specialists and emergency managers use Hazus to determine losses and the most beneficial 

mitigation approaches to take to minimize them. Hazus can be used in the assessment step in the 

mitigation planning process, which is the foundation for a community's long-term strategy to reduce 

disaster losses and break the cycle of disaster damage, reconstruction and repeated damage. Being ready 

will aid in recovery after a natural disaster.

As the number of Hazus users continues to increase, so do the types of uses. Increasingly, Hazus is being 

used by states and communities in support of risk assessments that perform economic loss scenarios for 

certain natural hazards and rapid needs assessments during hurricane response. Other communities are 

using Hazus to increase hazard awareness. Successful uses of Hazus are profiled under Mitigation and 

Recovery and Preparedness and Response. Emergency managers have also found these map templates 

helpful to support rapid impact assessment and disaster response.”

Additional notes:

A comprehensive sequence of Hazus training courses is available through FEMA’s Emergency 

Management Institute. On-line training is also available through the ESRI virtual training website at 

http://training.esri.com/gateway/index.cfm?fa=search.results&searchterm=Hazus. FEMA also provides 

on-line help desk support for Hazus users at https://hazus.msc.fema.gov.

Critical assessment questions/issues/dimensions: 

1. Comprehensiveness

a. Applicability across communities of varying size and type

The methodology and data sets cover the entire United States. A community can be 

defined as any combination of U.S. Census tracts. The model operates in a Geographic 

Information System (GIS) environment which provides map-based displays and supports 

spatial queries of the model inputs and outputs.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc.

Specifically applies to earthquake (EQ), flood (FL) and hurricane (HU) hazards.

c. Applicability across different recovery time scales

Focus is on immediate physical, economic and (to a lesser degree) social impacts, but the 

model does produce outputs on expected loss of use for buildings, loss of use for

infrastructure (EQ and FL only), shelter requirements, casualties (EQ only), building 
56

contents and inventory losses, lost wages and income and indirect economic losses (EQ 

and FL only)

d. Representativeness across different systems/interdependencies

Individual infrastructure items are assessed in the EQ and FL models; however systems 

interdependencies are not modeled. Economic interdependencies are considered in the 

indirect economic loss model.

2. Utility

a. User-friendliness – Basic operation requires familiarity with GIS software. Advanced 

operation requires detailed understanding on EQ, FL and/or HU hazards and the 

underlying “Inventory” data sets. The entire inventory data set is updated every 10 years 

following the release of new national Census data. Intervening updates to portions of the 

inventory data have been made for some releases.

b. Level of technical resources required to implement the methodology – Moderate. Many 

communities have individuals on staff with sufficient GIS expertise to operate Hazus at a 

basic level (Level 1) with little or no training. FEMA offers a series of free courses at the 

Emergency Management Institute for both basic and advanced training.

c. Value of methodology outputs in supporting resilience planning – Some value,

particularly in assessing a community’s current level of physical, economic and (to a 

lesser degree) social risk to EQ, FL, and/or HU hazards. Hazus can also be used to 

quantify the benefits of some mitigation activities.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? –

Hazus is broadly consistent with PPD-21, but it does not fully address recovery times or 

the full range of hazard types.

3. Ability to assess/measure/predict...

a. Physical impacts, interdependencies, and recovery times – Yes on physical impacts and 

recovery times, but not interdependencies.

i. Buildings (EQ, FL, and HU) – Yes 

ii. Transportation (EQ and FL only) –Yes, for individual highway, railway, light 

rail, bus, port, ferry, and airport system components, such as individual highway 

bridges or railway segments.

iii. Energy (EQ and FL only) – Yes, for individual system components, such as

generation plants and substations. Distribution circuits are aggregated and 

modeled at the census tract level. 

iv. Communications and Information (EQ and FL only) – Yes, for individual system 

components. In the EQ model, central offices and broadcasting stations are the 

only communication system components modeled.

v. Water and Wastewater (EQ and FL only) – Yes, for individual system 

components such as treatment plants, pumping stations, storage tanks, and 

transmission aqueduct segments. Distribution pipelines are aggregated and 

modeled at the census tract level.

b. Economic impacts and recovery times – Recovery times are explicitly included in 

economic loss estimates produced by the model, but the economic outputs are not 

tabulated or viewable as a function of time. While Hazus can be used to assess losses 
57

avoided through some mitigation measures, it does not assess costs and therefore does not 

produce estimates of return on investment.

c. Social impacts and recovery times – Hazus produces estimates of post-event shelter 

requirements (EQ, FL, HU) and casualties (EQ only). Demographic data on age, 

ethnicity, income, owner/renter, etc. from the U.S. Census are used in the shelter and 

casualty models. The weights applied to the various demographic factors used within 

these models can be modified by the user.

d. Ecological impacts and recovery times – No.

4. Which of the following techniques are used in the methodology?

a. Checklists – Optional

b. Interviews – Optional

c. Ratings – Yes

d. Existing national data sets – Yes

e. Physical inspections – Optional

f. Engineering analyses – Yes

g. Statistical inference – Yes

h. Simulations – Yes

5. Critical Assessment

a. Maturity – Hazus has been in use and supported by FEMA since 2004. Some portions of 

the model are incomplete (e.g., the hurricane model does not address lifelines, casualties, 

or indirect economic losses).

b. Uniqueness/innovativeness – The combination of publicly available default data sets, 

models, technical documentation and training is quite unique.

c. Objective and repeatable? – Yes

d. Scientific basis/merit/empirical evidence to support the method

i. Current/prior use of methodology – Widespread.

ii. Extent to which the approach been field tested – Numerous pilot studies, state 

and local mitigation plans, technical reports, and peer-reviewed publications are 

based on or include Hazus results.

iii. Extent to which the approach has been self-assessed – Technical oversight 

committees regularly reviewed the development of the Hazus EQ, FL and HU 

models. Some validation studies have been conducted and published, but more 

are needed.

iv. Has it been applied to real-world planning and management? – Yes.

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? Several pilot studies and some validation studies have been 

conducted and published, but more are needed.

vi. Clarity – Hazus outputs for individual scenarios are fairly well understood by 

users, but probabilistic outputs are frequently misinterpreted and misused. 
58

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? – N/A.

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach?

ii. What are the next steps that must be taken to improve the methodologies 

currently available?

There are significant gaps between the results produced by Hazus and the information 

required for a community-level resilience assessment methodology, particularly in the 

areas of interdependencies, social impacts and recovery times. But many of the Hazus 

methodologies and the types of results they produce could become portions of a larger 

tool.
59

12. Summary Assessment of Nine Existing Methodologies

The methodologies reviewed in Sections 3 through 11 were not necessarily developed specifically for the 

purpose of assessing community resilience, but each offers approaches, indicators, or processes that are 

relevant and potentially applicable to community resilience assessments. 

The methodologies were assessed on five broad dimensions: (1) scope, (2) utility, (3) impacts assessed, 

(4) techniques used, and (5) critical assessment. As discussed in Section 2, the scope dimension includes 

the breath of community sizes, hazard types and intensities, recovery time scales (e.g., short, medium, and 

long-term), systems (i.e., different components of the built, social, and natural environments), and system 

interdependencies covered by the methodology. The utility dimension addresses the clarity and ease-ofuse of the methodology, the extent of subject matter expert support required to implement the 

methodology, the value of the methodology outputs for community resilience planning, and how well the 

methodology and its outputs align with the definition of resilience given in PPD-21. The impacts assessed

dimension addresses the extent to which the methodology addresses the impacts of disruptive events and 

the ensuing recovery times for the built environment, economy, social systems or environmental systems. 

The techniques used dimension summarizes the data collection and analysis techniques that are either a 

core element or an optional element of the methodology. Finally, the critical assessment dimension 

considers the degree of maturity, innovativeness, objectivity, and scientific merit of each methodology. 

Each assessment was made in the context of community resilience planning and assessment, specifically 

as it pertains to planning decisions involving the built environment.

A summary assessment of the nine selected methodologies is provided in Figure 3. A separate legend is

shown to the right of each dimension to provide a general indication of each methodology’s relative 

strengths within that dimension. In some cases (e.g., where the methodology is in an earlier stage of 

development or where the factor being assessed is not clearly addressed in the documentation reviewed), 

a question mark is shown to indicate that additional information is required. The techniques used

dimensions is for informational purposes. These rows are not color-coded, as the types of techniques used 

are not considered, in and of themselves, methodology strengths or weaknesses.

None of the nine methods reviewed is uniformly strong in each of the first three dimensions. Therefore, 

even those models that were found to be relatively strong in the critical assessment dimension have

important limitations in terms of important aspects of community resilience, such as interdependencies 

and recovery times, or the breadth of community sizes or hazard types covered. As a result, a reasonable 

strategy is to attempt to combine the best features of several existing and emerging methodologies to 

produce a new community resilience assessment methodology that is specifically designed to address the

community planning needs identified in the NIST Community Resilience Planning Guide for Buildings 

and Infrastructure Systems.
60

Figure 3. Critical Assessment Summary of Nine Existing Community Resilience Methodologies
61

13. Concluding Remarks

Resilience is assessed in a variety of ways. Some assessment methodologies propose qualitative 

approaches while others use quantitative approaches, presenting the outcomes often in the form of 

scorecards or dashboards by measuring key resilience aspects. Such visual representations are often 

desirable as they can provide a direct and simple way of presenting the information both for experts in the 

field or for decision makers. In general, most of these methodologies focus on social issues, and in some 

cases, the focus is on one particular social service or system, such as in the case of THRIVE (Toolkit for 

Health and Resilience in Vulnerable Environments, see Appendix A), which focuses on health issues. An 

exception is the Argonne National Laboratory Resilience Index (see Appendix B); however, in this case,

consideration for social aspects is weak. As a result, it is relevant to indicate that one important issue 

found in most of these scorecard or dashboard methodologies is the relative weak integration of 

infrastructure resilience metrics into a social environment. 

In some cases where infrastructure is considered, such as the case of SPUR or the UNISDR Scorecard, 

the resilience targets selected, such as lifeline service restoration times, are not clearly explained or 

justified and do not correlate with existing practices, standards, and regulatory rules applied in each 

industry. For example, when applying the SPUR methodology, planners in San Francisco set a goal of 

having service restored to 90% of electricity power, water, wastewater and communication networks 

users by the third day after a design-level earthquake strikes the area. However, the origins of the 90% 

figure and the 3-day timeline are not clearly explained. Such performance targets would presumably be 

set in response to a particular community’s perceived needs and, thus, may not necessarily be directly 

applicable to other communities or other hazards. Moreover, if similar service restoration objectives are 

applied to widely different infrastructures, this would not recognize the existence of infrastructure 

dependencies that lead to different service restoration times for different infrastructure systems. This 

different restoration timeline was demonstrated in a paper by Dueñas-Osorio and Kwasinski2that showed 

through the study of data from the 2010 Chile earthquake that water and communication networks service 

restoration necessarily lag electric power service restoration.

In some cases, such as the Oregon Resilience Plan or SPUR, the methodologies focus on a particular 

hazard or geographic domain. As a result, there may be difficulties in extending these resilience 

frameworks to other zones or hazards. In addition, in those methodologies that do attempt to relate

infrastructure resilience to community resilience, the integration of infrastructure resilience issues with 

the community functions they support is generally weak. Such is the case of the Coastal Resilience Index 

which oversimplifies resilience as being directly related to the percentage of infrastructure damaged. For 

example, in many disasters, such as Hurricane Sandy, it is possible to observe that very little damage to 

electric power infrastructure could still produce very long power outages affecting most or all customers 

in a given area. One primary reason for this characteristic is the significant influence of human-based 

decision processes (e.g. logistics) on the restoration process. As a result, a moderate level of damage over

a very extensive area may take as long to restore as a far more intense level of damage over a relatively 

small area.

In addition to weak integration of infrastructure and social resilience aspects, there seems to exist 

considerable difficulty in balancing simplicity and accuracy. In part, this issue is caused by the difficulty 

of representing community resilience with a single metric or a small dimensional framework. Relatively 

simple resilience frameworks, such as CART, seem to be able to convey resilience assessments for 

decision makers in a straightforward way by presenting seven main attributes. However, difficulties can 

 

2 Dueñas-Osorio, L. and Kwasinski, A., “Quantification of Lifeline System Interdependencies after the 27 February 

2010 Mw 8.8 Offshore Maule, Chile Earthquake.” Earthquake Spectra, vol. 28, no. S1, pp. S581-S603, June 2012, 

http://dx.doi.org/10.1193/1.4000054
62

occur when decision-makers need to prioritize resources by considering the relative importance of each of 

the seven attributes. CART also represents an approach to collect data for the assessment based on 

questionnaires. Although such an approach is valid, its utility is constrained by the subjective views of the 

stakeholders answering the questionnaire, which often leads to a methodology that requires complex 

questionnaire development, administration and evaluation processes in order to ensure objectivity and 

reproducibility in the final outcome. The tradeoffs between simplicity through a single index developed 

from aggregating resilience-based metrics vs. objectivity is an issue discussed and considered in the 

Rockefeller Foundation’s City Resilience Framework as they develop their Community Resilience Index.

In general, resilience frameworks have not been validated or applied in a practical context, although some

recent programs, such as CARRI or the CRF, have been making efforts to fill this gap. Hence, significant 

efforts in data collection will need to be undertaken in order to provide the necessary inputs to validate 

new or existing methodologies. In parallel, substantial interdisciplinary research combining social and 

technical fields with data analytics will be necessary in order to achieve the goal of a fully integrated, yet 

simple, resilience methodology. 

In summary, the primary gaps observed in the methodologies reviewed include the following:

1. Weak integration of social systems and the built environment (e.g., dependencies among social 

and physical systems are generally not taken into account)

2. Lack of objective and repeatable methodologies for establishing existing system performance 

baselines and desired system performance targets

3. Limitations in the hazards or geographic domains considered

4. Difficulty in balancing simplicity and accuracy

5. Limited validation or practical application

At a secondary level, this review also includes aspects of “sector-focused” resilience methodologies, most 

notably energy, communication, and transportation infrastructure (Appendices B, C, and D, respectively). 

The motivation for also including these points-of-view is to assess how aspects of these domain-specific 

practices and philosophies can be incorporated or adapted for use in broader, community-based resilience 

assessments. The sector-focused methodologies tend to emphasize system-wide and facility-specific 

vulnerability. Concepts of recovery, social networks and interaction, and economics generally tended to 

receive considerably less attention. Not surprisingly, as engineering-oriented approaches, they also tended 

to be more quantitative and systematic in terms of the approaches used.

Assessments also vary considerably in terms of the expectations of the evaluator. While some are broad 

and general and can be completed by an evaluator without focused knowledge and experience, others 

require a much deeper understanding of operational and planning aspects of a community or a system to 

be able to accurately and effectively conduct the assessment. The review also revealed the level of input 

and output detail, utility, and applicability. While some were broad and shallow in terms of the data and 

information input requirements, others were much more detailed and required much greater and 

comprehensive data sets. Not unexpectedly, the output level of breadth and depth also commonly 

corresponds to these levels of input. This concept was also evident in the NOAA Coastal Resilience Index 

methodology wherein the resilience rating was used to suggest a time until restoration of basis community 

services. However, there were no criteria shown as to how this time to recovery was estimated.

An overarching finding within the energy, communication, and transportation sectors is that the concept 

of comprehensive integration of resilience needs and planning into routine practice is an emerging 

concept. While vulnerability assessment and design and planning for operational continuity and system 

robustness has been a growing part of engineering practice for some time, particularly since the 

September 11th terrorist attacks and Hurricane Katrina, the integration of resilience aspects of these 

systems into a more comprehensive community-wide resilience planning remains a relatively recent and 

emerging idea.
63

A. Annotated Bibliography of Additional Community-Based Assessment 

Methodologies

The critical review of community-based or community-centric assessment methodologies included 

available literature from both basic (fundamental) and applied social science research, as well as what can 

be referred to as community-based initiatives that incorporate dimensions of empirically-based social 

science research (e.g., Rockefeller 100 Resilient Cities, Zurich Flood Resilience Alliance). This literature 

was reviewed and assessed to gauge how tools, methods, philosophies, and approaches from these arenas 

may be applied to enhance community resilience. A focus of this effort was to search for practices that 

have been particularly effective or have shown potential promise for application in a variety of settings. 

The review of social science information sources revealed a range of foci and approaches, which strongly 

reflect: 

 The mission of the funding source(s) for the development and/or implementation of the effort or 

activity

 The intended purpose or use of the methodology

 The disciplinary background of the authors

Generally speaking, each methodology provides a working definition of resilience in the context of 

hazards and disasters, and begins with a discussion of the importance of community resilience. A number 

of the documents reviewed present detailed reviews of literature associated with disaster resilience, as 

well as related concepts that ground the work in empirically based social science research (e.g., BRIC, 

THRIVE, Building Community Resilience to Disasters: A Way Forward to Enhance National Health 

Security). All of the methodologies provide a set of dimensions (or categories) of community disaster 

resilience and, in many cases, include a list of indicators (variables) or potential indicators associated with 

each dimension.

In cases where the methodologies involve the engagement of community stakeholders, process-oriented 

guidelines for implementation are included (e.g., CARRI’s CRS, THRIVE, CART, Coastal Resilience 

Index). For methodologies that are heavily quantitative—typically involving extant data that are readily 

available—report or article authors provide appropriate details with respect to indicators used and 

strategies for data analysis and modeling (e.g., BRIC, ResilUS, The PEOPLES Framework). 

Most of the approaches provide information regarding the challenges and limitations of the 

methodologies, which is extremely useful in terms of replicability.

In addition to the nine methodologies reviewed in Sections 3-11, the following methodologies were 

identified in the social science research and community-based initiatives literature. These additional 

methodologies are summarized below in annotated form. The annotations are both evaluative and 

informative to summarize the information contained in the source as well as to highlight relative strengths 

and weaknesses of the idea relative to the objectives of the NIST Community Resilience Program. Where 

appropriate, the annotations also include notable, applicable, and potentially useful and interesting ideas, 

information, resources that could be applied within the context of general community planning. 

Toolkit for Heath and Resilience in Vulnerable Environments (THRIVE). 2004. Prevention 

Institute. 

Retrieved 12/09/14 at:

http://minorityhealth.hhs.gov/assets/pdf/checked/THRIVE_FinalProjectReport_093004.pdf
64

This report, funded by the Office of Minority Health (OMH), U.S. Department of Health and 

Human Services, focuses on communities of color to address health disparities. This methodology 

includes the following four clusters: 

o Built environment 

 Activity-Promoting Environment 

 Nutrition-Promoting Environment 

 Housing 

 Transportation 

 Environmental Quality 

 Product Availability

 Appearance/Ambiance

o Services and institutions

 Public Health, Health, and Human Services 

 Public Safety 

 Education and Literacy 

 Community-Based Organizations 

 Cultural/Artistic Opportunities

o Social capital

 Social Cohesion and Trust 

 Collective Efficacy

 Civic Participation/Engagement

 Positive Behavioral/Social Norms

 Positive Gender Norms

o Structural factors

 Ethnic/Racial Relations

 Economic Capital

 Media/Marketing

This approach was piloted in three locations: Hidalgo County, New Mexico; Del Paso Heights, 

Sacramento, California; and New York City District Public Health Offices (East Harlem, South 

Bronx, Central Brooklyn). Results suggest that this approach:

o Contributes to a broad vision about community health

o Enables systematic planning

o Is applicable in rural and urban settings

o Is useful to practitioners and community members

o Is an effective tool for strategic planning in organizations at the community level
65

The report also presents guidelines including:

o Sample actions

o Planning processes

o Resources

o Tools

o Community examples for each cluster

Additional notes:

- It is not clear whether THRIVE was implemented beyond the pilot sites

- The original set of indicators has been simplified and reduced to an assessment worksheet with 

twelve factors for use at a local level. See: 

http://www.preventioninstitute.org/component/jlibrary/article/download/id-801/127.html

- The new tool is primarily a checklist approach, intended to determine what areas need action 

and help communities to prioritize these

ResilUS: A Community Based Disaster Resilience Model. 2013. SB Miles and SE Chang. 

Retrieved 12/09/14 at:

http://www.tandfonline.com/doi/abs/10.1559/1523040638136#.VID6WLfwvAU

ResilUS is a spatial decision support tool based on measurable aspects of community capital. This 

simulation model operationalizes community resilience across multiple, hierarchical scales in 

relation to a range of policy and decision variables associated with each scale. ResilUS is 

implemented using fragility curves to model loss and Markov chains to model recovery with 

respect to time. The tool explicitly represents damage associated with a hazard event to three 

elements of community capital:

o The physical built environment

o Economics

o Personal (i.e., health)

ResilUS was applied to the 1994 Northridge earthquake disaster in order to calibrate several output 

variables with empirical data. The authors note several key limitations to the model:

o Representation of decisions and policies that is probably overly simplistic and limited

o The lack of a capability for modeling relocation of households within the study region

o The overall reliability and performance of the model across a range of disasters is unknown

o Some elements and outputs of the model simply could not be verified empirically, much 

less calibrated because of lack of empirical data

Additional work is underway by the authors to further evaluate and calibrate the model. In the 

meantime, they suggest that in its current form ResilUS is best suited for education, training, and 

public awareness purposes.
66

A Framework for Defining and Measuring Resilience at the Community Scale: The PEOPLES 

Resilience Framework. 2010. CS Renschler, AE Fraizer, LA Arendt, GP Cimellaro, AM Reinhorn, 

and M Bruneau. NIST Office of Applied Economics Engineering Laboratory. 

Retrieved 12/09/14 at:

http://www.esf.edu/glrc/library/documents/FrameworkforDefiningandMeasuringResilience_2010.p

df

This report describes a framework for defining and measuring disaster resilience at the community 

scale. Funded by the National Institute of Standards and Technology (NIST), this work builds on 

previous research by the Multidisciplinary Center for Earthquake Engineering Research (MCEER) 

linking four resilience properties (robustness, redundancy, resourcefulness, and rapidity) and 

resilience dimensions (technical, organizational, societal and economic). 

The authors present seven dimensions characterizing community functionality; these are 

represented by the acronym PEOPLES: 

o Population and Demographics

o Environmental/Ecosystem

o Organized Governmental Services

o Physical Infrastructure

o Lifestyle and Community Competence

o Economic Development

o Social-Cultural Capital

The PEOPLES Resilience Framework provides the basis for development of quantitative and 

qualitative models that measure continuously the functionality and resilience of communities 

against extreme events or disasters in any or a combination of the above-mentioned dimensions. 

According to the report, over the longer term, the intended use of these models is to enable the 

development of decision-support software tools to help planners, key decision makers and 

stakeholders enhance the disaster resilience of their communities.

The authors note: “The framework presented in this report uses as a central part in the definition 

and quantification of resilience, the basic functionality of various components contributing to 

community resilience. These functionalities are complex functions of various parameters, which 

need to be yet defined and quantified. Previous attempts of such quantifications indicate that there 

is still much to be done before the implementation of this concept is feasible and efficient. 

However, the initial framework defined in this report, can serve as guide for definitions of 

functionalities, parameters identifications, data collection, computational evaluations, etc.” (p. 45) 

Beginning on page 46 of the report, the authors discuss recommendations for future research.

Building Community Resilience to Disasters: A Way Forward to Enhance National Health Security. 

2010. Chandra, A., Acosta, J., Stern, S., Uscher-Pines, L., Williams, M., Yeung, D. , Garnett, J., & 

Meredith, L. RAND Corporation Technical Report. 

Retrieved 12/09/14 at:

http://www.rand.org/content/dam/rand/pubs/technical_reports/2011/RAND_TR915.pdf
67

The goal of this report is to provide a roadmap for federal, state, and local leaders who are 

developing plans to enhance community resilience for health security threats. Funded by U.S. 

Department of Health and Human Services Assistant Secretary for Preparedness and Response, it 

describes options for building community resilience in key areas, in the context of national health 

security. As described by the authors, “The report is intended principally for community leaders 

developing a local strategy for building resilience. These leaders include government and 

nongovernment actors who may be part of local emergency planning committees or related 

community planning teams. Given the limited evidence base on what activities are most effective 

for bolstering community resilience, the report is not intended as an implementation guide or “how 

to” toolkit.” (p. xiv)

The information presented in this report was generated by three activities: (1) a substantive 

literature review; (2) six stakeholder focus groups across the United States, and (3) three meetings 

with relevant subject matter experts (SMEs). Based on these, the report offers a set of eight “levers” 

for action and five community core components for building resilience. The eight levers are:

o Wellness

o Access

o Education

o Engagement

o Self-sufficiency

o Partnership

o Quality

o Efficiency

Wellness and access contribute to the development of the social and economic wellbeing of a 

community and the physical and psychological health of the population. Specific to the disaster 

experience, education can be used to improve effective risk communication, engagement and selfsufficiency are needed to build social connectedness, and partnership helps ensure that government 

and nongovernmental organizations (NGOs) are integrated and involved in resilience-building and 

disaster planning. Quality and efficiency are ongoing levers that cut across all levers and core 

components of community resilience.

The five core components of community resilience, in the context of national health security, are:

o Physical and psychological health of the population

o Social and economic well-being of the community

o Effective risk communication information for all populations including at-risk populations

o Integration and involvement of organizations in planning, response, and recovery

o Social connectedness for resource exchange, cohesion, response, and recovery

The report includes recommendations for ways to strengthen community resilience and identifies 

implementation challenges.

Additional notes:

- The authors write that both the CART (Community Advancing Resilience Toolkit) and the 

work by the Community and Regional Resilience Institute (CARRI) are “Important community 

tools [that] have been developed to assist communities in enhancing aspects of resilience, and 
68

they should be used.” (p. 2). They go on to state that their report is specific to meeting the needs 

of national health security.

- The authors list three key limitations for readers to consider: (1) the activities in this report 

reflect only those identified in focus group and SME discussions; (2) the list of activities was 

not subject to additional vetting based on feasibility or the sociopolitical context in which 

activities may be implemented; and (3) none of these activities has undergone rigorous 

evaluation. Before a community resilience toolkit can be developed, communities will need to 

use this roadmap, report on lessons learned, and assess the impact of implementing particular 

activities.

Canterbury Wellbeing Index. 2014. Canterbury Earthquake Recovery Authority. 

Retrieved 12/09/14 at:

http://cera.govt.nz/sites/default/files/common/canterbury-wellbeing-index-june-2014-fulldocument.pdf

The Canterbury Wellbeing Index was developed by the Canterbury Earthquake Recovery Authority 

(CERA) with the support of multiple agencies to track the progress of the social recovery in greater 

Christchurch following the Canterbury Earthquake sequence. In late 2011, CERA convened a series 

of meetings with representatives of 28 agencies to identify the social indicators that should be 

tracked through the recovery. Expert advice was received through the literature review of 

international best practice “Designing indicators for measuring recovery from disasters”, 

undertaken by Canterbury District Health Board.

Indicators are used to identify emerging social trends and issues to enable agencies to respond in a 

timely way. The Canterbury Wellbeing Index is also prepared to provide the greater Christchurch 

community with accurate and comprehensive information about the social recovery.

The Index includes indicators that address: 

o Knowledge and skills

o Economic well-being

o Housing

o Health

o Safety

o Social connectedness

o Mental wellbeing

Operationalizing Resilience against Natural Disaster Risk: Opportunities, Barriers, and a Way 

Forward. 2014. Keating et al. Zurich Flood Resilience Alliance and the International Institute for 

Applied Systems Analysis. 

Retrieved 12/09/14 at:

http://www.iiasa.ac.at/web/home/research/researchPrograms/RiskPolicyandVulnerability/Resilienc

e-lowres_2.pdf

Focusing on flooding hazards, this report suggests that resilience can be:

o Defined by distinct properties
69

o Operationalized through an Iterative Risk Management (IRM) process that links experts 

risk analysis with stakeholder involvement

o Measured at a certain point in time and over time

The authors propose a set of resilience metrics that includes:

o Physical capital

o Social capital

o Human capital

o Financial capital

o Natural capital

To date, the process has been applied to at least seven international sites, although there do not 

appear to be any reports available on the outcomes of doing so.

Characteristics of a Disaster Resilient Community - Department for International Development 

(DFID) Interagency Group. 2007. J Twigg. Developed for the DFID Disaster Risk Reduction 

Interagency Coordination Group. 

Retrieved 12/09/14 at:

https://practicalaction.org/docs/ia1/community-characteristics-en-lowres.pdf

The intent of this report is to show the characteristics of a disaster resilient community. Based on 

the UNISDR (United Nations International Strategy for Disaster Reduction) work, the themes 

presented include:

o Governance

o Risk assessment

o Knowledge and education

o Risk management and vulnerability reduction

o Disaster preparedness and response

Each theme is divided into three sections: 

o Components of resilience

o Characteristics of a resilient community

o Characteristics of an enabling environment

This is primarily a qualitative approach on the front end, undertaken collaboratively by experts and 

community leaders. During implementation, the process evolves to include quantitative measures 

where available and appropriate.
70

This Page Intentionally Left Blank
71

B. Methodologies Considered for Electric Power Infrastructure 

Resilience

As a subtask within the broader community resilience literature review process, a sampling of 

methodologies and tools within the specific area of electric power infrastructure (EPI) resilience was also

identified and reviewed. This appendix provides a brief overview of the EPI resilience literature reviewed 

and a more detailed review of a single EPI methodology following the format used in Sections 3-11.

Although most of the sources for resilience metrics in EPI systems represent academic perspectives, the 

evaluation also considers industry-based groups of electric power distribution utilities and work 

originated at one of DOE’s National Labs. The specific EPI papers and reports reviewed are listed below:

[EPI-1] J. A. Momoh, S. Meliopoulos and R. Saint “Centralized and Distributed Generated Power 

Systems - A Comparison Approach Future Grid Initiative White Paper,” PSERC Publication 

12-08, June 2012.

[EPI-2] P. Stockton, “Resilience for Black Sky Days,” Report of the National Association of 

Regulatory Utility Commissioners, 2014, 2.

[EPI-3] A. Al Majali, A. Viswanathan and C. Neuman, “Analyzing resiliency of the smart grid 

communication architectures under cyber attack,” in Proc. CSET'12 Proceedings of the 5th 

USENIX conference on Cyber Security Experimentation and Test, 2012, pp. 1 -4.

[EPI-4] Eric D. Vugrin and Jennifer Turgeon, “Advancing Cyber Resilience Analysis with 

Performance-Based Metrics from Infrastructure Assessments.” International Journal of Secure 

Software Engineering (IJSSE), vol. 4, no. 1, pp. 1-22, July 2012.

[EPI-5] Y. Wei, C. Ji, F. Galvan, S. Couvillon, G. Orellana, and J. Momoh, “Non-Stationary Random 

Process for Large-Scale Failure and Recovery of Power Distributions,” arXiv.org, vol. 

abs/1202.4720, 2012.

[EPI-6] C. Miller, M. Martin, D. Pinney, and G. Walker,” Achieving a Resilient and Agile Grid,” The 

National Rural Electric Cooperative Association, April 2014, http://www.nreca.coop/wpcontent/uploads/2014/05/Achieving_a_Resilient_and_Agile_Grid.pdf. 

[EPI-7] R. Bent, “Grid Resilience: Design and Restoration Optimization,” Los Alamos National Lab, 

LA-UR-14-25832, presented at IEEE PES General Meeting, 2014.

[EPI-8] Lt. Col. G. A. Montoya, “Assessing Resilience in Power Grids as a Particular Case of Supply 

Chain Management,” M.S. Thesis, Air Force Institute of Technology, Dayton, OH, March 

2010.

[EPI-9] M. Ouyang and L. Dueñas-Osorio, “Resilience model and simulation of smart grids,” in 

Proceedings Structures Congress, 2011, Las Vegas, NV, 1996-2009.

[EPI-10] M. N. Albasrawi, N. Jarus, K. A. Joshi and S.S. Sarvestani, “Analysis of Reliability and 

Resilience for Smart Grids,” in Proc. IEEE 38th Annual Computer Software and Applications 

Conference (COMPSAC), July 2014, pp. 529 – 534.
72

[EPI-11] Office of the State of Maryland Governor M. O’Malley, Weathering the storm. Report of the 

Grid Resiliency Task Force, Sept. 2012.

[EPI-12] A. Kwasinski, “Field technical surveys: an essential tool for improving critical infrastructure 

and lifeline systems resiliency to disasters,” in Proc. IEEE 2014 Global Humanitarian 

Technology Conference, Oct. 2014, pp. 1-7.

[EPI-13] Fisher, R.E., et al. Constructing a Resilience Index for the Enhanced Critical Infrastructure 

Protection Program, Report Argonne National Laboratory # ANL/DIS-10-9, Aug. 2010.

In some cases, the methodology reviewed is representative of a broader set of similar methods that focus 

on a particular aspect of EPI systems resilience, such as works in cyber-security related resilience.

General Assessment

Traditionally, the electric power industry has been a heavily regulated business in which top-down

approaches to operations and planning ensure that the system can be operated within certain performance 

(or, as it is called, power quality and reliability) standards within some acceptable financial and economic 

objectives. In this traditional environment, electric power distribution reliability metrics typically exclude 

major event days in order to provide a common reference for all utilities irrespective of where they are 

located or which hazards they may suffer. This top-down approach is representative of how traditional 

electric power grids are built and operated; i.e., the traditional electric power grid planning and operations 

paradigm is based on a mostly centralized architecture in which a small number of relatively large 

generators are interconnected over large areas to serve many much smaller loads located in many 

communities or neighborhoods. 

In the past few years, however, electric power grids have been experiencing an important transformation 

towards what has been called smart grids. Although several of the technologies associated with smart 

grids, such as smart meters, present little benefits in terms of resiliency during natural disasters, other 

technologies, such as distributed generation (microgrids) present significant resilience advantages 

compared to conventional power grids. These technologies can be applied at the individual community 

level and, thus, this technological development shifts the focus from the described top-down approach 

into a bottom-up approach in which individual electricity users or communities can set their own electric 

power supply performance goals and metrics. Thus, in general, the electric power infrastructure (EPI) 

technologies that present the most significant advantages in terms of resilience from a community 

perspective are typically those that imply a distributed power generation, control and distribution 

architecture, i.e., the opposite paradigm of conventional power grids. Such drastic system planning and 

design paradigm changes have important implications in many aspects of the operation of electric power 

systems, including how their performance and, thus, their resilience is measured. A key consequence of 

the new paradigm in modern smart grids is an increased need for performance standards developed from 

the bottom-up by taking into account the electricity users and their individual community’s needs and not 

necessarily the service level that electric power utilities can provide. As a result, individual customers

could develop their own performance objectives based on their needs and constraints (mostly of the 

economic type).

The resilience metrics proposed in the literature reviewed tend to follow either the top-down or bottom-up

paradigm in EPI design and planning. In general, resilience metrics or assessments prepared by 

government agencies or utilities, such as [EPI-11] or [EPI-2], follow a top-down approach in which 

resilience is related to established system reliability metrics and other regulatory benchmarks. In this topdown approach, resilience tends to follow the traditional definition used in the electric power industry 

focusing on service restoration speed only. Nevertheless, [EPI-2] tends to fit better into the PPD-21 

definition of resilience by proposing an approach that does not exclude major event days when evaluating 
73

power grid resilience. Still, [EPI-2] does not considers the effect of human activities and decisions on 

power grids resiliency, nor does it consider the effects of power grids dependencies on other

infrastructures. Other proposed metrics that only consider the service restoration component of resilience 

are presented in [EPI-5] and [EPI-10]. In terms of applicability, both [EPI-5] and [EPI-10] have a mostly 

theoretical approach that may be difficult to implement by utilities or electricity users. Another work that 

follows a primarily theoretical approach is [EPI-1]. One merit of [EPI-1] is that it also considers 

distributed generation systems. However, like [EPI-10], [EPI-1] does not consider the effects of 

dependencies, and it only considers the service restoration component in its resilience metric. Hence, 

none of these works account in their resilience metrics for how well the system withstands a disruptive 

event. Another theoretical approach is discussed in [EPI-8] which relates resilience assessments to supply 

chain management. Although [EPI-8] presents a novel and interesting approach, its application seems 

complex and possibly outside the expertise of electric power utilities personnel or electric power users in 

a distributed generation system. The approach presented in [EPI-9] is also theoretically-based, and it 

considers three components for resilience metrics that are relevant within the definition of resilience in 

PPD-21: resistance, absorption and recovery. However, the complexity of the metrics in [EPI-9] make 

them difficult for electric power utilities to adopt and apply. Other theoretical approaches focus on

specific technological aspects of resilience. This is the case of [EPI-3] and [EPI-4] which represent a 

much larger group of publications that considers the effect of cyber-security on resilience.

Other works have followed a different bottom-up approach, by focusing on resilience from the user’s

perspective. Some of these approaches consider the effects of dependencies and human decisions and 

actions on electric power infrastructure resilience. Both of these aspects are considered in [EPI-12], which 

proposes a metric equivalent to one of the reliability indices defined in electric power distribution 

reliability standards with the addition of major event days in the resilience quantification. This index is 

the average service availability index (ASAI), which seems to represent the definition of resilience in 

PPD-21 well because it depends both on the time power is provided to customers and the time it takes to 

restore service when a failure happens. Extending the use of established reliability indices in order to 

measure resilience, as is done in [EPI-11] and [EPI-2], seems to be advantageous, as it simplifies the 

adoption into planning and design processes. A work that acknowledges the importance of human factors 

at the same level as physical components is [EPI-6], but their proposed resilience assessment is qualitative

rather than quantitative. Yet another resilience metric that considers human-driven processes as integral 

part of the system is the metric proposed in [EPI-7]. This resilience framework has the added benefit of 

being based on the PPD-21. Another equally important aspect of [EPI-7] is that resilience metrics are 

user-driven. Hence, it can be applied in the new EPI environment based on the paradigm of more 

distributed, user-oriented systems. 

Several research organizations, both private and government, have been increasingly focusing on power 

grid resilience. Among the private ones, the Electric Power Research Institute (EPRI) and the Edison 

Electric Institute have been working on technologies to improve power supply resilience. Several DOE 

National Labs, such as Lawrence Berkeley National Lab and National Renewable Energy Lab have also 

been studying technologies to improve power supply resilience, particularly through microgrids. Argonne, 

Sandia, and Los Alamos National Labs have been involved in efforts to assess resilience. Argonne

National Lab, in [EPI-13], has a somewhat similar approach to that in [EPI-9], as it also considers 

resilience based on three key attributes: robustness, resourcefulness, and rapid recovery. These attributes 

contribute to produce a combined resilience metric for each critical infrastructure (including EPI). 

Although the method is intended to be replicated, its reliance on questions and answers from stakeholders 

in order to adjust a weight factor adds a level of complexity that is not present in other approaches. 

Ensuring accuracy in the question and answer process requires a large number of participants with a level 

of rigor, training and expertise difficult to find in a practical setting at a community level. Sandia National 

Lab is participating in the Rockefeller Foundation’s framework under a five-year partnership announced 

in April 2014. In the case of Los Alamos National Lab, their proposed approach is the one that has been 

discussed as part of [EPI-7]. 
74

Towards the future, the most significant need is to harmonize user-based resilience requirements and 

metrics with electric power utilities capabilities and regulation. This need is not only with respect to

resilience metrics but also a much larger set of issues related to the evolution of power distribution and 

control architectures into more distributed systems with more control, planning and design relevance to

users. Hence, it is expected that significant research, standardization activities and regulation actions will 

be required related to smart grids and power systems technologies which will necessarily include a deeper 

look into resilience issues. In terms of community resilience, the increased reliance on power grids for 

financial services, transportation (e.g. electric vehicles), communications, health services, etc., will 

motivate a more integrated view of infrastructure as a system-of-systems in which understanding 

dependencies will play a critical role in order to improve community resilience. Moreover, it has recently 

been noted that 40% of the EPI industry workforce will be eligible for retirement within the next 5 years. 

Therefore, a deeper look into how human-driven processes affect EPI resilience should lead to 

interdisciplinary studies of electric power systems planning and operation as well as increased efforts in 

electric power education.

Among the approaches reviewed for measuring resilience of electric power infrastructure, [EPI-7] seems 

to be among the most relevant in terms of broad assessments of community resilience. Part of this 

relevance originates in its bottom-up view that places a focus on user perspectives. This is an important 

aspect of [EPI-7] because it can be argued that community resilience metrics should be more 

representative of the electricity user’s perspective than of an electric utility’s ability to comply with topdown performance metrics developed by regulatory agencies. Moreover, [EPI-7] considers the need to

account for new technological approaches for improved electric power supply resilience, such as 

microgrids, that are expected to be the type of solutions implemented at community levels. Yet another 

important aspect of [EPI-7] is that it considers the influence of humans on electric power infrastructure 

resilience. These factors combine to merit a more detailed discussion of [EPI-7] below.

Assessment of a Specific Methodology for Electric Power Infrastructure Resilience: R. Bent, Los 

Alamos National Lab [EPI-7]

1. Comprehensiveness

a. Applicability across communities of varying size and type

Not applicable to communities. However, the proposed framework can be used to 

evaluate the effect of EPI systems resilience on a broad set of communities of different 

types and with varying size.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc.

Yes. 

c. Applicability across different recovery time scales

Yes.

d. Representativeness across different systems/interdependencies

The general philosophy could be applied in other systems, particular the proposed 

resilience design process. However, some other aspects, such as user-defined resilience 

metrics could be difficult to apply in other contexts.

2. Utility

a. User-friendliness. 

Somewhat. Although the general process and approach is simple, some of the analysis 

and calculations in some of the steps may be complex. Still, the basic metric considered 
75

in the description of the method – i.e., based on maximum number of affected users vs. 

outage duration – is a simple approach.

b. Level of technical resources required to implement the methodology. 

It depends on the level of detail the evaluation wants to achieve and the resilience metrics 

chosen by users. In most cases, it is expected that the metric, such as the described one of 

number of outages vs restoration time, will be simple to require few resources.

c. Value of methodology outputs in supporting resilience planning. 

High. The proposed metric can be directly applied to planning activities in a broad set of 

applications.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? 

Yes, fully. The entire framework is based on PPD-21 objectives.

i. Resilience = “The ability to prepare for and adapt to changing conditions and 

withstand and recover rapidly from disruptions.”

ii. Includes the ability to withstand and recover from deliberate attacks, accidents, 

or naturally occurring threats or incidents

3. Ability to assess/measure/predict... 

a. Physical impacts, interdependencies, and recovery times.

i. Buildings – No

ii. Transportation – Maybe in the long term

iii. Energy – Yes

iv. Communications and Information – Maybe in the long term

v. Water and Wastewater – No

b. Economic impacts and recovery times – Yes, but not applied to the social aspects listed 

below but, instead, within the context of EPI systems.

i. Employment

ii. Revenues

iii. Return on investment

c. Social impacts and recovery times – Not applicable to this aspect.

i. Health (physical and mental)

ii. Shelter

iii. Safety

d. Ecological impacts and recovery times – Not applicable to this aspect.

4. Which of the following techniques are used in the methodology?

a. Checklists

b. Interviews 

c. Ratings 

d. Physical inspections
76

e. Exercises (e.g., tabletop)

f. Engineering analyses – Yes

g. Statistical inference – Yes

h. Simulations – Yes

i. Quantification based on power grids operation and design models.

5. Critical assessment

a. Maturity. Limited. Still needs significant development.

b. Uniqueness/innovativeness. Quite innovative.

c. Objective and repeatable? Yes it is objective and repeatable. However, accepting userdefined metrics may lead to a large number of metrics that may be difficult to consider in 

an integrated way.

d. Scientific basis/merit/empirical evidence to support the method. Yes. The analysis is 

based on common experiences and observations in electric power industry.

i. Current/prior use of methodology. Somewhat but in a different context. In the 

present environment, methodologies tend to be prescribed by a regulatory agency 

from the top-down. This proposed approach is, instead, from the bottom-up 

because metrics are selected by users. 

ii. Extent to which the approach been field tested. On a very limited basis. The tests 

performed in microgrids in Japan considering multiple quality levels or the Kita 

Kyushu test bed may represent initial tests of the philosophy in this approach.

iii. Extent to which the approach has been self-assessed. Little to none.

iv. Has it been applied to real-world planning and management? No.

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? No.

vi. Clarity. Extremely clear.

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? Not applicable. This is 

not a community-level metric.

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? In terms of 

interdependencies, more multidisciplinary research is needed to characterize and 

model interdependencies. More interdisciplinary research is also needed in order 

to consider the effect of human aspects on EPI resilience. Such an

interdisciplinary approach will likely require collaboration between electrical 

engineers, civil engineers, industrial engineers, economists, sociologists and 

psychologists. Impact of the proposed approach on regulatory action and electric 

utilities operation need to be further assessed.

ii. What are the next steps that must be taken to improve the methodologies 

currently available? Improve interdependency modeling and develop theories that 

could quantify the effect of human processes on resiliency in an integrated way.
77

Evaluation

A review of resilience metrics for EPI systems suggested two opposing approaches. While [EPI-2] 

represents the established top-down approach based on regulatory action, [EPI-7] has a bottom-up 

approach that may seem to be more relevant considering the paradigm changes in electric power industry 

towards more distributed systems. Such bottom-up approach tends also to support technologies that 

enhance resilience at a local level, such as microgrids, and, thus, tend to be more supportive of EPI 

resilience metrics more applicable at community levels. Moreover, [EPI-7] follows the PPD-21 definition

of resilience. 
78

This Page Intentionally Left Blank
79

C. Methodologies Considered for Information and Communication

Infrastructure Resilience

As a subtask within the broader community resilience literature review process, a sampling of 

methodologies and tools within the specific area of information and communication infrastructure (ICI) 

resilience was also identified and reviewed. This appendix provides a brief overview of the IPI resilience 

literature reviewed and a more detailed review of a single IPI methodology following the format used in 

Sections 3-11. The sources for resilience metrics in ICI systems include communications industry-driven 

groups, academic centers or individuals, and government agencies. The specific IPI papers and reports 

reviewed are listed below:

[ICI-1] University of Manchester, RABIT – the Resilience Assessment Benchmarking and Impact 

Tool. Available online http://niccd.org/node/77

[ICI-2] L. Strigini, Resilience Assessment and Evaluation of Computing Systems . Chapter 1, Fault 

Tolerance and Resilience: Meanings, Measures and Assessment, K. Wolter et al. (eds.), 

Springer-Verlag Berlin Heidelberg 2012

[ICI-3] A. van Moorsel et al. “AMBER Assessing, Measuring, and Benchmarking Resilience,” Project 

Report FP7 – 216295, June 2009, available online at 

https://eden.dei.uc.pt/~rbarbosa/files/md_242_amber_d2.2_stateoftheart_v2.0final_submit.pdf

[ICI-4] Federal Communications Commission, “Vulnerability Assessment and Feasibility of Creating 

a Back-Up Emergency Communications System,” Report to Congress , January 2008, 

available online at http://transition.fcc.gov/pshs/docs/clearinghouse/case-studies/ECSvulnerability-assessment-report.pdf

[ICI-5] Federal Communications Commission, “In the Matter of Improving the Resiliency of Mobile 

Wireless Communications Networks Reliability and Continuity of Communications Networks, 

Including Broadband Technologies,” FCC 13-125, September 2013.

[ICI-6] A. Kwasinski, “Field technical surveys: an essential tool for improving critical infrastructure 

and lifeline systems resiliency to disasters,” in Proc. IEEE 2014 Global Humanitarian 

Technology Conference, Oct. 2014, pp. 1-7.

[ICI-7] P. Cholda et al., “Quality of Resilience as a Network Reliability Characterization Tool,” IEEE 

Network, vol. 23, no. 2, pp. 11-19, March 2009.

[ICI-8] ITU-T Focus Group on Disaster Relief Systems, Network Resilience and Recovery “Technical 

Report on Telecommunications and Disaster Mitigation,” Technical report version 1.0, June, 

2013, available online at http://www.itu.int/en/ITUT/focusgroups/drnrr/Documents/Technical_report-2013-06.pdf

[ICI-9] J.P.G. Sterbenz, E. K. Cetinkaya, M. A. Hameed, A. Jabbar, and J.P. Rohrer, “Modeling and 

analysis of network resilience,” in Proc. 2011 Third International Conference on 

Communication Systems and Networks (COMSNETS), pp. 1-10.
80

[ICI-10] P. Trimintzios, “Measurement Frameworks and Metrics for Resilient Networks and Services: 

Technical report,” European Network and Information Security Agency (ENISA), February 

2011.

General Assessment of Methodologies Considered for Information and Communication 

Infrastructure Resilience

Information and communications infrastructure systems span a broad range of technologies including, but 

not limited to, wireline, wireless and CATV networks, public broadcasting radio and TV, and data storage 

and processing systems. As traditional circuit-based communication systems migrate into packet-based 

(IP) systems, these individual systems are increasingly evolving into a single integrated infrastructure in 

which technologies and devices at the end user level have more influence on perceived resilience. This 

evolution into more distributed end-user focused packet-based ICI systems creates new challenges in how 

their resilience can be measured. These challenges originate in the fact that performance evaluations of 

packet-based ICI systems based on quality of service consider a continuous scale in which performance 

degradation is accepted, whereas traditional evaluation of circuit-based systems generally considers a 

discrete assessment of whether a communication can be established or not. The metric approach 

explained in [ICI-7] provides an extensive discussion of the various ways in which resilience (or 

performance) is evaluated in various parts of ICI systems and proposes to extend the concept of quality of 

service into a quality of resilience, which is considered in [ICI-10] as part of the proposed European 

metric for ICI systems resilience. One merit of the approach in [ICI-7] is its simplicity. However, it may 

be difficult to have the ICI industry to accept such a metric because it is an indicator that has not been 

validated whereas other indicators, such as availability, has been used and validated for several decades of 

operation of ICI networks. 

Availability seems to be the most widely accepted and recognized performance metric for ICI networks,

and it is the metric suggested in [ICI-6] and [ICI-10] to quantify resilience. One advantage of availability 

used for this purpose is that it matches well with the concept of availability in PPD-21 because 

availability depends on the expected time a system is operating—i.e. it measures withstanding capabilities 

to natural disasters—and on the expected time a system is not operating—i.e., it measures service 

restoration speed. Since availability is a well-accepted and widely used metric, it can also be used in order 

to perform probability risk assessments or to be applied in other tools that translate performance 

expectations into cost requirements as part of ICI mitigation or network deployment plans.

In addition to the aforementioned quality-based approach in [ICI-7], [ICI-9] also considers performance 

degradation as a possible operating condition. However, the approach discussed in [ICI-9] is a theoretical 

approach with a complicated application. Similar complexity issues are found in other academic methods, 

such as [ICI-2]. The FCC represents the other end of the “spectrum” in terms of scientific basis in the 

analysis supporting the proposed approach and complexity. Although the FCC methodologies [ICI-4, ICI5] have a direct and simple application, their fundamentals are not based on a scientific study. This is 

understandable as the goal of FCC documents is regulation and legislative action support. Therefore, 

other factors influence the process leading to proposing approaches to evaluate resilience that in some 

cases is qualitative. One other issue with the top-down regulatory approach found in [ICI-4, ICI-5] is the 

resistance found by key ICI industry players in their application which contrasts another governmentdriven framework presented in [ICI-10], which was developed with a bottom-up approach and with a 

broad participation from the European ICI industry.

The influences of human aspects and ICI dependency on other infrastructures are key factors affecting 

network resilience during extreme events. Both of these factors are generally recognized in the resilience 

evaluation that have been considered. However, with the exception of [ICI-6], none of the resiliency 

assessment methods provide a way to quantify the degree of dependency or to relate it with resilience. In 

terms of considering human aspects, several methodologies—particularly those considering availability as 

a resilience metric—are able to identify the influence of human factors in order to improve resilience but 
81

there is still a gap in quantitatively representing how human decision and management processes affect 

resiliency metrics. 

Among the approaches reviewed for measuring resilience of information and communication 

infrastructure, [ICI-10] seems to be among the most relevant in terms of broad assessments of community 

resilience. Part of this relevance originates in its development as a bottom-up approach and with broad 

participation from the European ICI industry. The framework also seems to present a good balance in 

achieving a resilience measurement approach that includes all application domains in ICI systems, and the 

use of availability as a metric fits well with the definition and view of resilience in PPD-21. These factors 

combine to merit a more detailed discussion of [ICI-10] below. 

Assessment of a Specific Methodology for Information and Communication infrastructure 

Resilience: ENISA Resilience Framework [ICI-10]

1. Comprehensiveness

a. Applicability across communities of varying size and type

Not applicable to communities. However, the proposed framework can be used to 

evaluate the effect of ICI systems resilience on a broad set of communities of different 

types and with varying size.

b. Applicability across hazards of different type, intensity, geographic extent, duration, 

warning time, etc.

Yes. 

c. Applicability across different recovery time scales

Yes.

d. Representativeness across different systems/interdependencies

In general, yes. Although such characteristics are not discussed in the ENISA report, 

availability can be used for other systems and can also be used to measure the degree of 

dependence on other infrastructures. The concept of availability may not be applicable 

when evaluating quality of service or performance degradation-type of characterization. 

However, these characterizations are typically used for entertainment services, such as 

streaming video, that are rarely a service priority or even a need during natural disasters 

or other extreme events.

2. Utility

a. User-friendliness. Yes, very much.

b. Level of technical resources required to implement the methodology. It depends on the 

level of detail the evaluation wants to achieve. However, there are many techniques, such 

as minimal cut set approaches, and commercially available tools that simplify 

calculations even in very complex settings.

c. Value of methodology outputs in supporting resilience planning. High. The proposed 

metric can be directly applied to planning activities.

d. Does the method define/evaluate resilience in a manner that is consistent with PPD-21? 

Yes, fully. Availability is measured by dividing the “up-time” to the sum of the “up-time” 

and “down-time.” Up-time measures withstanding characteristics whereas down time 

measures recovery speed.

3. Ability to assess/measure/predict... 
82

a. Physical impacts, interdependencies, and recovery times.

i. Buildings – No

ii. Transportation – No

iii. Energy – No

iv. Communications and Information – Yes. Although the ENISA report does not

use the proposed metric to evaluate dependencies, it has been shown that 

availability can be used to measure the degree of dependencies.

v. Water and Wastewater – No

b. Economic impacts and recovery times – Yes, but not applied to the social aspects listed 

below but, instead, to down time costs and associated recovery times in ICI systems.

i. Employment

ii. Revenues

iii. Return on investment

c. Social impacts and recovery times – Not applicable to this aspect.

d. Ecological impacts and recovery times – Not applicable to this aspect.

4. Which of the following techniques are used in the methodology?

a. Checklists

b. Interviews – Yes

c. Ratings 

d. Physical inspections

e. Exercises (e.g., tabletop)

f. Engineering analyses – Yes

g. Statistical inference – Yes

h. Simulations

i. Quantification based on well-established methods measuring network availability

5. Critical assessment

a. Maturity – Substantial

b. Uniqueness/innovativeness The application of availability to measure resilience is 

moderately innovative. However, availability metrics have long be used in 

communication systems.

c. Objective and repeatable? Yes. The approach is completely objective as it is based on 

well-known reliability theory, probability, statistics and math principles. As a result, it is 

fully repeatable. 

d. Scientific basis/merit/empirical evidence to support the method. Yes, complete. There are 

decades of data of network operators using availability metrics that provides an extensive 

empirical evidence of its applicability.

i. Current/prior use of methodology. Yes, the method is based on well-known 

metrics that have been in used in the ICI industry for a considerable time.
83

ii. Extent to which the approach been field tested. Extensive as per the previous 

response.

iii. Extent to which the approach has been self-assessed. Little to none.

iv. Has it been applied to real-world planning and management? Yes. Availability is 

a fundamental metric that has been used in ICI systems planning and 

management. 

v. Does evaluative information exist regarding how the methodology has worked in 

different settings? Yes.

vi. Clarity. Extremely clear.

vii. If the method combines or aggregates component- or system-level metrics into 

one or more community-level metrics, what is the theoretical or empirical basis 

for how the lower-level metrics are scaled and combined? Not applicable. This is 

not a community-level metric.

e. Have gaps of knowledge, data, analytical approaches regarding resilience metrics been 

identified?

i. How can we address the gaps in the metrics and approach? In terms of 

interdependencies, more multidisciplinary research is needed to characterize and 

model interdependencies. More interdisciplinary research is also needed in order 

to consider the effect of human aspects on ICI resilience. Such interdisciplinary 

approach will likely require collaboration between electrical engineers, civil 

engineers, industrial engineers, economists, sociologists and psychologists. 

ii. What are the next steps that must be taken to improve the methodologies 

currently available? Improve interdependency modeling and develop theories that 

could quantify the effect of human processes on availability in an integrated way.

Evaluation

The framework developed by ENISA seems to present a good balance in achieving a resilience 

measurement approach that includes all application domains in ICI systems. Moreover, the framework 

was developed from the bottom-up based on substantial input from stakeholders, particularly from the 

communications industry. Use of availability as a metric also fits well with the definition and view of 

resilience in PPD-21. Moreover, since availability is a well-established performance and planning metric 

used in communication networks, its adoption and application should be simple and direct. 
84

This Page Intentionally Left Blank
85

D. Methodologies Considered for Transportation Infrastructure 

Resilience

General Findings

As part of the critical review of community resilience assessment methodologies, literature pertaining to 

the resilience of transportation systems, assets, networks, resources, and evaluation methods was assessed 

to gauge how tools, methods, philosophies, and approaches from the transportation field, specifically, 

may be applied to enhance community resilience, more broadly. Another component of the review was to 

search for practices that have been particularly effective or have shown potential promise for application 

in a wider context. 

The review of transportation information sources showed that discussions of resilience practice tended to 

be categorized based on three primary considerations, including:

 the mode of transportation and specific roles, needs, and characteristics of it; 

 the type of hazard/incident/event/condition that had occurred or is anticipated; and/or 

 the level of detail or broadness in scope or scale of the assessment. 

In terms of modal focus, a considerable amount of resilience-related discussion was associated with 

efforts for the assessment and evaluation of freight movements. A particular focus of this work was on the 

application and contribution of transportation systems and assets for recovery and resupply after major 

disruptive events. Especially notable was the emphasis on the disruption and connectivity loss within 

highway networks and, to a somewhat lesser extent, capacity disruptions of maritime ports and intermodal 

freight terminals. Another prominent area of work was on topics related to public transportation. 

Literature related to public transport modes included the use of transit for evacuation before an event, 

then as a means to support recovery and repopulation efforts after disasters. Although not a mode per se, 

another major area of resilience efforts in transportation was in the protection and recovery of critical 

transportation infrastructure, most notably bridges and tunnels. Numerous sources presented emerging 

and theoretical methods for the assessment and ranking of vulnerability and need for strengthening to 

resist more extreme events/conditions. The amount of work in this area is not surprising because of the 

perceived susceptibility of these assets to terrorist acts. Infrastructure protection also supports efforts to 

improve the general condition of bridges and pavements which has been in decline throughout the U.S. in 

recent decades. 

Resilience assessments in the transportation literature also tended to be categorized and discussed in terms 

of hazard type or based on certain conditions that have occurred or could occur in an area. Although 

emergency management typically seeks an all-hazards approach to planning, many of the assessment 

discussions referenced needs and preparations to counter fairly specific conditions. The most common 

were conditions associated with global climate change and sea level rise. Other common themes were 

enhanced security for terrorist threats and other event-specific natural hazards known to occur or likely to 

threaten specific areas. Most notably among these were hurricanes, floods, snow storms, and earthquakes. 

Within each of these hazard conditions a common approach was to assess resilience in terms of 

infrastructure vulnerability or exposure, commonly focusing on security and identifying past system 

failures. 

The discussion within the documents identified in the search also ranged fairly widely in terms of the 

level of depth and detail in the analyses and evaluations. Based on this, they could also be categorized on 

whether they were higher-level, policy or conceptual discussions or whether they were geared more 

toward specific application. In many of the documents in the former group, authors tended to treat the 

topic of resilience in theoretical terms and presented planning and evaluation needs at strategic levels. 

Works of this type tended to be academic/scholarly treatments and examined long-term, broad-based 
86

benefits that could be gained from integration these ideas into practice. Those that were more focused at a 

tactical level were oriented toward operations and practices and were most detailed with regard to supply 

chains and freight and shipping logistics and delivery topics. A notable exception to this was the 2013

New York City study described in the next section which names specific projects and discusses plans for 

their implementation and assessments of their benefits from multiple hazard and response perspectives. 

More detail was typically included on topics related to roadway construction and maintenance, especially 

those associated with global climate change and sea level rise. 

Another differentiator of resources identified in the review was whether the ideas had been applied in 

actual practice, were just suggested or proposed for application, or were presented merely as ideas and 

theories for discussion from which broad ideas and concepts could be incorporated into future resilience

support efforts. Generally speaking, a large percentage of the resources and publications that were found 

were more theoretical or proposed rather than had been applied. Many of them, particularly those in 

freight and commercial delivery operations, tended to leverage systems engineering techniques such as 

optimization methods and numerical analysis techniques to reroute vehicles and minimize travel delays 

and disruptions. There were also several examples that featured modeling and simulation to evaluate 

alternative ideas. 

The topic of resilience in construction and maintenance has grown significantly in recent years and, in 

particular, in the wake of Hurricane Sandy in 2012 and the 2014 North Georgia snow event. Interestingly, 

maintenance planning and practices also seems to be developing in a bottom-up direction since needs for 

snow plowing, flood control, and other extreme weather events directly impact local agencies budgets 

first and these needs are perceived to be growing in terms of both regularity and severity.

A common technique to examine network robustness was via the use of modeling and simulation (at 

various levels of fidelity) to assess impacts of losses in network connectivity and the implementation of 

alternate routing strategies. Interestingly, some of the techniques were also extended to employing the 

concept of system resilience for disasters to serve needs during planned events and lower-level incidents. 

Under conditions like traffic crashes or sporting events, concerts, etc. some level of system functionality 

could be lost or additional capacity or alternative routing may be necessary from several hours to as much 

as several days. Interestingly, it has also been suggested that concepts that increase and/or assure 

resilience could even be extended to during routine daily peak periods in which surge traffic volume 

creates conditions that warrant additional system capacity and adaptive/creative approaches to problem 

solving. 

Although the formal discussion of resilience seems to be a recent development in transportation, many 

resilience activities have been taking place as a matter of routine course for decades. Often, resiliencerelated work is not viewed in a broader context or formally identified as such or coordinated into an 

overall framework. It is suggested that this has limited the overall of such efforts. To be most effective, 

the literature suggests that efforts should be planned in coordination with an overall integrated plan of 

resilience-related activities. Another idea suggested in the literature was the need to test, evaluate, and 

continually improve and update resilience improvement activities. This can be done using models and 

simulation as well as partial- and full-scale exercises and drills.

In the sections that follow, the most relevant and appropriate documents that were identified in the 

transportation literature are summarized in annotated form. The annotations are both evaluative and 

informative to summarize the information contained in the source as well as to highlight relative strengths 

and weaknesses of the idea relative to the NIST CRAM effort. Where appropriate, the annotations also 

include notable, applicable, and potentially useful and interesting ideas, information, resources that could 

be applied within the context of general community planning. 
87

Annotated Bibliography

United States Department of Transportation (USDOT). 2012. FHWA’s Climate Change and Extreme 

Weather Vulnerability Assessment Framework. Federal Highway Administration. Washington, 

DC. 

https://www.fhwa.dot.gov/environment/climate_change/adaptation/publications_and_tools/vulner

ability_assessment_framework/fhwahep13005.pdf

This is a guide for transportation agencies to assess vulnerability to climate change and extreme 

weather events. The guide gives an overview of key steps to conduct vulnerability assessments 

and examples to demonstrate ways to gather and process information. The assessment is 

conducted within a three step process that includes:

o defining study objectives and scope; 

o assessing vulnerability; and

o incorporating results into decision-making.

The objectives for vulnerability assessments include: 

o siting new assets in areas less vulnerable to climate change;

o educating staff regarding overall climate risks to the agency’s transportation system; and

o informing the development of adaptation strategies. 

These objectives are used to select and characterize relevant assets and identify climate variables 

for the study of climate change and extreme weather vulnerability within a transportation context. 

Vulnerability assessment tasks include: 

o gathering and integrating data and information on asset location, characteristics, and 

climate sensitivities; 

o gathering and obtaining information on historical weather events and projected climate; 

o combining the asset and climate information to identify vulnerabilities; and

o potentially, assigning a level of risk of the climate impacts on the assets. 

The Guide suggests that assessment should involve an iterative process in which information 

gathered on assets may inform climate information needs and vice versa.

Victoria Transport Policy Institute (VPI). 2014. Evaluating Transportation Resilience - Evaluating the 

Transportation System’s Ability To Accommodate Diverse, Variable and Unexpected Demands 

With Minimal Risk. Victoria, BC, Canada. http://www.vtpi.org/tdm/tdm88.htm

This evaluation discussion is a chapter of the VPI Transportation Demand Management (TDM) 

online Encyclopedia. This resource defines resilience as a system’s ability to provide its critical 

functions under variable, uncertain and extreme conditions and involves identifying a system’s 

critical functions, its vulnerabilities, and ways to reduce vulnerabilities. It suggests that analysis 

should involve more than simply contingency planning (considering “what would happen if...”) 

because it is not possible to predict every possible future condition. Below are specific steps in 

planning for resilience:

Define the System

The first step in this planning process is to define the extent of the system to be evaluated. For a 

transportation system this may include the transportation facilities and services in a particular 

jurisdiction or region, including those that connect outside of that area. All components of that 

system should be considered, including, for example, the pedestrian system, freight and package 

delivery systems, transportation for public services such as road maintenance and garbage 

collection, air travel systems, etc.
88

Identify Critical Functions

This involves identifying the transportation activities and services that are most valued to society. 

The list below is an example of “Basic Accessibility.”

1. Emergency response (police, fire, medical services, disaster relief, etc.).

2. Public services (utility repair and maintenance, garbage collection, etc.).

3. Freight and package delivery.

4. Commercial and business travel.

5. High value personal errands (medical appointments, basic shopping, etc.).

6. Commuting (travel to work and school).

7. Lower-value personal errands (social trips, recreational shopping, etc.).

8. Other low-value travel (leisure travel, cruising, etc.).

Identify Vulnerabilities

This involves identifying various ways that a system’s components and requirements could fail or 

become inefficient. Below are some examples of potential problems to consider.

o A network link is broken, such as a blocked sidewalk, bridge or roadway.

o A service fails, such as a bus strike, or a motorist loses his or her ability to drive.

o A group of users has difficulty walking or is unable to speak the local language.

o A critical resource becomes scarce and expensive, such as a petroleum shortage.

o A common source of information fails or provides false information, such as an incorrect 

announcement of travel conditions by a radio station.

o A particular official, technician or repair crew is unavailable during a crisis.

o A disaster requires emergency transport of a large number of people, many who cannot 

drive, and some with medical problems.

o A disaster causes extreme traffic congestion on a particular roadway.

o A particular service is discontinued due to inadequate demand, such as local bus or 

freight delivery services.

Identify Ways to Increase Resilience and Security

Find ways to reduce specific vulnerabilities, and incorporate Resilience principles into the 

planning and management of critical components of the transportation system. Below are 

examples of strategies that can increase Resilience:

o Increase transportation system Diversity. Ensure that there are opportunities for people to 

walk, cycle, rideshare, car share and travel by transit.

o Increase network redundancy and connectivity (e.g., the number of roads and transit 

routes in an area).

o Increase facility design and construction standards to withstand extreme conditions.

o Improve systems to identify potential problems, including physical damage, unusual 

demands and new risks.

o Improve the ability to communicate with transportation system users, including people 

with special needs, even under unusual conditions.

o Establish ways to Prioritize transportation system resources (road space, fuel, vehicle 

capacity) so it is available first to higher-value transportation activities.

Contingency-Based Planning

“Contingency Based” (or Responsive) planning refers to the idea that the planning process must 

be able to change over time in response to future needs. This involves the following steps:

1. Identify objectives (general things that you want to achieve) and targets (specific things 

that you want to achieve).
89

2. Identify various strategies that can help achieve the objectives and targets. These can 

include both projects that increase capacity and demand management strategies.

3. Evaluate the costs and benefits of each strategy (including indirect impacts, if any), and 

rank them according to cost-effectiveness or benefit/cost ratios.

4. Implement the most cost-effective strategies as needed to achieve the stated targets. 

5. After they are implemented, evaluate the programs and strategies with regard to various 

performance measures, to ensure that they are effective.

6. Evaluate overall results with regard to targets to determine if and when additional 

strategies should be implemented

Contingency-based planning addresses uncertainty by deploying solutions on an as-needed basis. 

For example, a transportation plan may identify 5 strategies to implement immediately, another 4 

to implement in two years if stated targets are not achieved, and another 3 can be implemented 

further in the future if needed. This tends to be cost effective and flexible, because strategies are 

only deployed if they are needed, and additional strategies can be ready for quick implementation 

if unexpected changes create additional needs. This approach is ideal for medium and long-range 

transport and land use planning. 

Hillsborough County, Florida Metropolitan Planning Organization (MPO). 2014. Hillsborough County 

MPO: Vulnerability Assessment and Adaptation Pilot Project. Prepared by Cambridge 

Systematics, Inc., Jacobs Engineering Group, Inc., and Florida Atlantic University. Tampa, FL.

http://www.planhillsborough.org/wp-content/uploads/2013/10/NoAppendix_HillsboroughMPO_FHWA-Pilot-Final-Report.pdf

This study is one of 19 Pilots across the country conducted under the Federal Highway 

Administration’s (FHWA) second-round climate change vulnerability assessment program, and 

was funded in part through a FHWA grant. The objective is to identify cost-effective strategies to 

mitigate and manage the risks of coastal and inland inundation for incorporation into the 

Hillsborough County MPO’s 2040 Long Range Transportation Plan (LRTP), into the County’s 

Post Disaster Redevelopment Plan (PDRP), and into transportation planning and decision-making 

processes more generally. 

This study is comprised of three primary technical phases:

Phase 1: This phase includes the assembly of a countywide inventory of multimodal 

transportation assets. This phase also entailed development of potential future coastal and inland 

inundation scenarios, and an assessment to identify existing or planned transportation assets 

potentially at-risk from sea level rise (SLR), storm surge (Categories 1 and 3), and inland 

flooding.

Phase 2: The next Phase utilized the MPO’s travel demand model to estimate the losses in 

regional mobility associated with disruption of those facilities.

Phase 3: The final technical phase focused on the estimation of general economic losses 

associated with the disruption of selected critical links, using the Regional & Economic Models, 

Inc. (REMI) tool, and the development of strategies for managing potential climate risks 

(adaptation investments). This phase concluded with the calculation of basic measures of the

potential cost-effectiveness (net avoided losses) of an illustrative package of adaptation strategies 

vs. a no adaptation (business as usual) scenario.

The assessments returned two summary variables that describe the relative cost effectiveness of 

the illustrative adaptation strategy package proposed for each asset: 1) Estimated net 

benefits/avoided losses resulting from reductions in the duration of disruption (expressed in 

dollars), and 2) the “tipping point,” the number of days of avoided disruption required for the 
90

strategy package to achieve cost neutrality. Both metrics are summarized below (estimated net 

benefits reflect the Category 3 surge scenario, which is common to all five assets).

This study provides a foundation for future assessments, which could adopt a variety of 

approaches, including a more granular, engineering-based focus on a specific facility or corridor 

(perhaps paired with a full-fledged Benefit/Cost Analysis), a more robust exploration of inland 

flooding issues, or a longer-term assessment horizon, for example. Future assessments will 

benefit from a substantial amount of emerging federal, state, and regional research—and, as with 

the current effort—should continue to leverage broad coalitions of multidisciplinary partners. 

Nagurney, A. 2102. Building Resilience into Fragile Transportation Networks in an Era of Increasing 

Disasters. Presentation to the 90th Annual Meeting of the Transportation Research Board. 

Washington, DC. http://supernet.isenberg.umass.edu/visuals/TRB_Panel_Nagurney_Talk.pdf

This resource is a PowerPoint presentation that discusses resilience from the perspective of broad 

terminology, history, characteristics, network vulnerabilities, needs, etc. The author presents a 

node and link methodology to model “system robustness” in terms of system optimization and the 

effect of lost links, diminished capacity, congestion, etc. The presentation also discusses climate 

change and the role of transportation in disaster relief and humanitarian relief supplies. However, 

no evaluation criteria or processes are applied, proposed, or discussed.

Cybulski, J.D. 2013. Transportation Infrastructure Resiliency: A Review of Transportation Infrastructure 

Resiliency in Light of Future Impacts of Climate Change. Graduate Research Paper. Volpe Center 

Report Number DOT-VNTSC-14-02. Harvard University. Cambridge, MA. 

http://www.bv.transports.gouv.qc.ca/mono/1149029.pdf

This resource is a Graduate Research Paper that focuses on physical infrastructure and provides 

illustrations of the level of potential risk. Specifically, it discusses infrastructure upgrades to 

counter climate change, including:

o Reconstruct and upgrade streets to include resiliency features;

o Integrate climate resiliency features into future street reconstruction projects including 

allowing surface run off from streets to soak into the ground rather than flow into the 

sewer system

o Installing bioswales and/or pre-cast permeable gutters, and adding or raising bulkheads to 

help prevent street flooding;

o Locate traffic signal electronics above flood levels and include emergency generators;

o Install floodgates and raise entrances to flood vulnerable traffic tunnels;

o Integrate climate change resiliency into routine planning and project development where 

it was not previously considered.

Proposed upgrades to help resume normal transportation operation to:

o Have a back-up emergency transit system in place when major modes are inundated;

o Work with a wide range of transportation agencies and other stakeholders around the 

region to identify the critical elements of the surface transportation network that need to 

be available quickly following different types of events;

o Implement high occupancy vehicle protocols for emergency situations to avoid vehicular 

traffic congestion;

o Plan for and install new pedestrian and bicycle facilities to improve connectivity to key 

transportation hubs;

o Update and construct new ferry landings, as well as purchase new ferries that can tolerate 

more extreme weather conditions;

o Improve communication from government agencies to the general public about 

transportation shut downs and restorations.
91

Proposed transit-specific initiatives to enhance climate resiliency, include:

o Expand the cities bus networks and bus priority corridors to accommodate commuters 

when subway lines are unavailable;

o Expand key ferry services

o Include more information and proposals on protecting property and preserving land.

This work also cites the USDOT - FHWA Climate Change & Extreme Weather Vulnerability 

Assessment Framework (USDOT 2012) – cited above.

Amdal, J. R. and S.L. Swigart. 2010. Resilient Transportation Systems in a Post-Disaster Environment: A 

Case Study of Opportunities Realized and Missed in the Greater New Orleans Region, 2010. 

University of New Orleans Transportation Institute. Paper 5. New Orleans, LA.

http://scholarworks.uno.edu/unoti_pubs/5/

This report is a theoretical discussion of resilience. The work focuses on lessons learned and 

opportunities missed in post-Katrina New Orleans and discusses the roles of key stake holders in 

the aftermath of this event. While no evaluation criteria or processes are applied, proposed, or 

discussed, numerous key needs and concepts are discussed. Among the key conclusions drawn 

from the research findings are:

o Achieving transportation resilience is an ongoing and incremental process.

o Progress towards transportation resilience within specific modes has been achieved 

although the network as a whole lacks communication and coordination across modes.

o Recently executed Memoranda of Understanding (MOUs) allow for maximum utilization 

of federal assets to support local resources in times of disaster.

o Metropolitan Planning Organizations (MPOs) are now recognized as a key facilitator to 

all levels of government for the recovery of critical transportation infrastructure based on 

their political and technical networks and their extensive regional data.

Ortiz, D.S., L. Ecola, H. H. Willis. 2009. Freight Transportation Resilience - How System-Wide 

Perspective Can Help Metropolitan Planning Organizations and Departments of Transportation. 

Published in Adding Resilience to the Freight System in Statewide and Metropolitan 

Transportation Plans: Developing a Conceptual Approach. National Cooperative Highway 

Research Program Project 8-36, Task 73. Transportation Research Board. Washington, D.C. 

http://www.rand.org/pubs/external_publications/EP51309.html

This report is a conceptual discussion of freight transportation system resilience. The authors 

discuss ways to view and, in a conceptual way, “measure” the resilience of transportation systems 

in a broad manner. The discussion is framed with the concepts of robustness, engineering 

resilience, and ecological resilience to measure resilience. The authors propose building on 

previous efforts that model where, when, and how system(s) degrade by developing a “time 

history of a disruption to a transportation network and the performance of the system as it 

responds to a disruption.” Although no specific evaluation criteria are proposed, a number of key

planning and operational actions are discussed. 

Pant, S. B. 2012. Transportation Network Resiliency: A Study of Self-Annealing. MS Thesis. Department 

of Civil and Environmental Engineering, Utah State University. Logan, Utah. 

http://digitalcommons.usu.edu/etd/1434

The objective of this research was to create a conceptual framework to quantify resilience and 

quantitatively discuss the properties determining resilience of transportation networks. The 

concepts are applied to a test network to illustrate the mathematical procedures and how they can 
92

help decision makers analyze relative improvements in resiliency as a consequence of proposed 

project alternatives and perform benefit-cost analyses. The method seeks to account for:

o Spare capacity of the network

o Network route diversity 

o Alternative mode availability

o Network management

o Network performance measures

 Travel time

 Emergency response time

 Resource availability

 Maintenance prioritization

The research evaluates aspects of resilience and under conditions of:

o Removing links

o Adding links

o Adding modes.

Bekkem, K., T.M. Adams, and E.T. Duran. 2011. Evaluating Operational Resiliency of a Highway 

Corridor, using a GIS-based Freight Network. National Center for Freight & Infrastructure 

Research & Education. Department of Industrial and Systems. University of Wisconsin, Madison, 

WI.

http://www.gis-t.org/files/g54I3.pdf

This resource is a PowerPoint presentation that discusses a theoretical assessment/framework 

perspective of resilience. The presentation uses major weather events (snow storms and floods) in 

Wisconsin to examine freight movement. Vulnerability of bridges, culverts and roadways and 

alternate routing options are examined under threats of:

o Flooding 

o Scour

o Overload volume

o Snow/Ice

o Tornado

The figure below summarizes the Wisconsin state-wide resilience plan and the author’s overview 

of vulnerability assessment. Resiliency values are defined as a function of economic value of the 

commodity flow, added vehicle miles traveled due to detours, and a risk priority number (RPN), 

where RPN is defined by corridor segment on a scale of 1 to 10 and averaged of over the threats 

and vulnerabilities listed above.
93

Zhang, L. and M. Jin. 2009. Framework Of Calculating The Measures Of Resilience for Intermodal 

Transportation Systems. Department of Industrial & Systems Engineering, Mississippi State 

University. Starkville, MS. 

http://ncit.msstate.edu/PDF/reports_59.pdf

This report presents a theoretical assessment to develop a framework for a measures of resilience 

(MOR) calculation for intermodal transportation systems in response to disasters. It also evaluates

the effectiveness of strategies for improving the MOR. Although freight transportation resilience 

is the primary focus of the report, passenger transportation is also considered as a part of an 

integrated transportation system. For the MOR calculation procedure, TransCAD was used to 

model the research area network and generate transportation data. A series of resilience indicators 

in terms of mobility, accessibility, and reliability were selected to evaluate the intermodal system 

performance based on the TransCAD outputs. A case study of the Mississippi Gulf Coast region

was used to demonstrate the effectiveness of the proposed MOR calculation procedure. 

Ip, W.H. and D. Wang. 2011. “Resilience and Friability of Transportation Networks: Evaluation, 

Analysis and Optimization. “ IEEE Systems Journal, Vol 5, No 2. pp. 189 - 198.

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5688189&tag=1

This paper is a theoretically-oriented discussion. The research focuses on rail lines and railway 

networks in the 20 largest cities in China. The authors apply a computational resource 

optimization approach for a link and node resiliency analysis to show the locations of “weak links 

and critical components of the railway networks.” The authors conclude that: “distributed hubs 

will have lower ‘friability’ than centralized ones,” where friability is defined by the authors as 

“the reduction in total resilience upon removing an edge or hub city.” No evaluation criteria or 

processes are applied, proposed, or discussed in actual real-life applications. However, the 

authors suggest that their procedure “may provide general guidance to those involved in the 

design of transportation networks.” In addition to the resilience and friability evaluation, a 

computational algorithm optimization model for network design is also recommended. 
94

Lavrenz. S. 2012. Resiliency Planning in Transportation: Practical Software Tools and Long-Term 

Visions. Iowa State University, Institute for Transportation. 

http://www.mtmug.org/Presentations/ResiliencyPlanning.pdf

This source presents theoretical work focusing on the need for and concepts of resiliency. In 

addition to describing resiliency planning in terms of needs and benefits, the author also describes 

and compares software tools available to transportation professionals for resiliency assessments. 

Comparisons are based on relative strengths and weaknesses, ease of use, input requirements, 

output applicability.

The author states that Criticality Accessibility Recoverability Vulnerability Espyability 

Redundancy, version 2 (CARVER2) is easy to use, free, and publicly available (with restrictions). 

It was developed by NI2 Center for Infrastructure Expertise and ranks infrastructure elements by 

threat of disruption and resulting effects, The ranking is done in terms of raw score, can be used 

for dissimilar infrastructure elements and can be used to assess likelihood in terms of 

infrastructure vulnerability. The Transportation Routing Analysis Geographic Information System

(TRAGIS) was developed by Oak Ridge National Laboratory and seeks to find the most efficient 

geographic routings for highway, rail, and water. It replaces HIGHWAY, INTERLINE models 

and is currently undergoing updates and minor redesign. The last model that was assessed in this 

source was the National Energy and Transportation Sustainability, Cost, and Resiliency for the 

21st Century (NETSCORE21). This system can be used to identify long-term investment 

strategies for energy and transportation systems, including highway transportation, conventional 

rail, high-speed rail, waterway, air passenger and freight movement. It can also be used for 

energy generation technologies, transmission and storage.

The source also listed several resilience metrics in terms of robustness and flexibility. These 

metrics are reproduced from the source and shown below:

Tamvakis, P. and Y. Xenidis (2012). “Resilience in Transportation Systems.” Procedia - Social and 

Behavioral Sciences 48, 3441 – 3450.

http://www.sciencedirect.com/science/article/pii/S1877042812030510

This source is a theoretical discussion of resilience concepts. It discusses the basic parameters of

resilience engineering and analyses, at a high level. It identifies critical factors in the resilience 

engineering of transportation systems to recover from sudden and severe stresses in a dynamic 

environment. Based on the review of the respective literature, this paper concludes with a general 

framework for resilience engineering of transportation systems.
95

NOAA Office for Coastal Management. 2014. NOAA Port Tomorrow - Resilience Planning Tool.

http://coast.noaa.gov/port/?redirect=301ocm

The Port Tomorrow: Resilience Planning Tool is an online resource that includes three tools for 

planning and evaluating the resilience of:

o Marine Transportation

o Port Communities

o Coastal Hazards

The tools include geospatial data and resources, checklists of resilience factors to consider, and 

links to local maps and stories used to illustrate the concepts. The tool was intended for use by 

professionals involved in infrastructure planning for ports and surrounding communities and 

those responsible for freight-related infrastructure project development or review. It does not 

include evaluation information per se’, but does include information of what to be prepared for 

(hazards), help with identifying resources, and anticipated impacts on society. It also includes an 

extensive annotated bibliography of sources of related information. 

Federal Highway Administration, 2014. Resilience Pilot Descriptions. United States Department of 

Transportation (USDOT), Washington, DC.

https://www.fhwa.dot.gov/environment/climate_change/adaptation/ongoing_and_current_researc

h/vulnerability_assessment_pilots/2013-2014_pilots/index.cfm

This webpage summarizes ongoing USDOT pilot project activities related to transportation 

climate resilience. The goal of the program is to give systematic consideration of climate change 

vulnerability and risk in transportation decision making, at the system and project level. It is 

based on Executive Order 13653 on preparing for climate impacts (November 1, 2013) and 

directs agencies to:

o Remove barriers, reform programs to promote climate resilience

o Provide data and tools for climate preparedness and resilience

Key products from the projects are updated engineering manuals, methods and processes; and key 

activities include engineering assessments related to: 

o Gulf Coast (Mobile, AL)

o Hurricane Sandy Follow-up and Vulnerability Assessment & Adaptation Analysis

o Transportation Engineering Approaches to Address Adaptation and Resiliency

o Climate Resilience Pilots

Projects are in progress in the locations listed below:

o Tennessee Department of Transportation (TDOT)

o Michigan Department of Transportation (MDOT)

o Capital Area Metropolitan Planning Organization (CAMPO) the MPO Austin TX region

o North Central Texas Council of Governments (NCTCOG)

o Maine Department of Transportation (Maine DOT)

o Arizona Department of Transportation (ADOT)

o Alaska Department of Transportation & Public Facilities (ADOT&PF) and Alaska 

Federal Land Management Agencies (FLMAs)

o Connecticut Department of Transportation (ConnDOT)

o Massachusetts Department of Transportation (MassDOT)

o Minnesota Department of Transportation (MNDOT)

o New York State Department of Transportation (NYSDOT)

o Metropolitan Transportation Council (MTC)

o Broward Metropolitan Planning Organization Palm Beach, Broward, Miami-Dade, and 

Monroe Counties in South Florida. 
96

o Oregon Department of Transportation (ODOT)

o California Department of Transportation (Caltrans)

o Hillsborough Metropolitan Planning Organization (Hillsborough MPO)

o Washington State Department of Transportation (WSDOT)

o Iowa Department of Transportation (IDOT)

o Maryland State Highway Administration (SHA)

Kafalenos, R., 2014. Gulf Coast 2 - FHWA Climate Resilience Framework (and Pilots). United States 

Department of Transportation, Washington, DC.

http://onlinepubs.trb.org/onlinepubs/conferences/2014/AssetManagement2014/Kafalenos%20-

%20Gulf%20Coast.pdf

This source provides an overall program summary of the pilot projects described above and 

assesses vulnerability from climate inputs, asset data, criticality, sensitivity to compute 

vulnerabilities and risk. Four slides from the Kafalenos presentation are reproduced below:
97
98

United States Department of Homeland Security (DHS). 2014. DHS Resiliency Evaluation Research. 

Washington, DC. https://www.dhs.gov/science-and-technology/evaluation-research

This is a website resource describing the mission and purpose of DHS research efforts in 

resiliency evaluation. As stated on its webpage, the research:

is used to assess the people, processes, tactics, techniques, procedures and 

policies necessary for implementing systems. It quantifies how effective the 

design of a system is as well as how the system performs in comparison to its 

goals and objectives. The research also highlights unintended consequences, 

the level of efficiency needed to maximize operational impact and potential 

uses for homeland security activities. Evaluation research applies rigorous 

and repeatable methods and involves:

• Identifying and engaging with key stakeholders

• Determining evaluation needs

• Developing relevant measures

• Collecting data

• Analyzing data

• Developing operationally relevant conclusions

• Sharing information

Although no specific papers or reports were listed, the website indicated that their evaluation 

research “encourages the use of scientific research in evidence-based homeland security 

decisions. This approach minimizes waste by providing a quantitative evaluation based on facts 

rather than emotions.”

Freckleton, D, K. Heaslip, W. Louisell, and J.Collura. 2012. “Evaluation of Resiliency of Transportation 

Networks After Disasters.” Transportation Research Board of the National Academies, Research 

Record No. 2284, Washington, D.C. pp. 109-116. 

http://trb.metapress.com/content/20012km345791888/
99

This paper provides a theoretical discussion of resilience. It defines many of the key high level 

concepts and terms in resilience and its evaluation. It presents a conceptual framework for the 

evaluation of transportation system resilience. Among its contributions are a suggestion of key 

input variables and measures. It uses a theoretical case example of a seismic event occurring 

along a major fault line on the Wasatch Front near Salt Lake City, Utah. From this, it computes 

the performance of a number of indicators and proposes areas for improvement.

Volpe National Transportation Systems Center. 2014. Transportation System Resilience, Extreme 

Weather, and Climate Change. Cambridge, MA.

http://www.volpe.dot.gov/events/transportation-system-resilience-extreme-weather-and-climatechange

This link points to a series of video presentations by recognized experts to discuss challenges, 

opportunities, and fresh approaches related to pressing issues in transportation resilience. The 

series, entitled “Transportation System Resilience, Extreme Weather, and Climate Change”, 

discusses broad ideas of the vulnerabilities of global transportation infrastructure, including the 

escalating threats of changing climate. Speakers are primarily from government agencies and 

authorities from across the US. 
100

This Page Intentionally Left Blank