# Indicator-based resilience assessment for critical
infrastructures – A review

Zhuyu Yang, Bruno Barroca, Alexandre Weppe, Aurélia Bony-Dandrieux, Katia Laffréchine, Nicolas Daclin, Valérie November, Khaled Omrane, Daouda Kamissoko, Frederick Benaben, et al.

► To cite this version:

Zhuyu Yang, Bruno Barroca, Alexandre Weppe, Aurélia Bony-Dandrieux, Katia Laffréchine, et al. Indicator-based resilience assessment for critical infrastructures – A review. Safety Science, 2023, 160, pp.106049. 10.1016/j.ssci.2022.106049. hal-03948697

HAL Id: hal-03948697

https://imt-mines-albi.hal.science/hal-03948697

Submitted on 20 Jan 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Review

# Indicator-based resilience assessment for critical infrastructures – A review

<figure><img src="image_2.png" /></figure>

Check for
updates

Zhuyu Yang<sup>a,b,*</sup>, Bruno Barroca<sup>a</sup>, Alexandre Weppe<sup>c</sup>, Aurélia Bony-Dandrieux<sup>c</sup>, Katia Laffréchine<sup>a</sup>, Nicolas Daclin<sup>c</sup>, Valérie November<sup>b</sup>, Khaled Omrane<sup>d</sup>, Daouda Kamissoko<sup>d</sup>, Frederick Benaben<sup>d</sup>, Hélène Dolidon<sup>e</sup>, Jérôme Tixier<sup>c</sup>, Vincent Chapurlat<sup>c</sup>

ARTICLE INFO

Keywords:

Resilience assessment

Indicator-based assessment

Critical Infrastructures

Resilience indicator

A B S T R A C T

A Safety Science focus on research in the science and technology of human and industrial safety. Critical In-
frastructures (CIs), as a crucial part of urban technological systems, are highly correlated with urban safety
management in terms of their resilience when cities are facing a crisis or disaster. According to many studies,
indicator-based resilience assessment has been used frequently to manage CIs in recent decades. Defining and
characterising indicators can be useful for managers of human and industrial safety, as it could help monitor and
improve the capacities and performance of CIs.

In recent years, critical infrastructures (CIs) have been damaged with increasing frequency (and will be so in
the future) following natural or technological disasters. CIs, such as buildings, transportation networks and
power systems, play an indispensable role in our society due to their importance for maintaining critical societal
functions, economic organisation and national defence. Therefore, the popularity of “critical infrastructure
resilience” has exploded in both academic and policy discourses. Indicator-based assessment is a convenient and
common tool to help understand, analyse and improve CI resilience in the scientific field. This paper produces a
state-of-the-art review of the existing indicator-based assessment of CI resilience. After a terminology presen-
tation, which helps clarify the objective of this study, this paper will show: 1) two methods for selecting the
current scientific papers applying indicators to assess CI resilience; and 2) analysis of the indicators in these
papers based on the study objective. The results show that there are many indicators and they do not have a
uniform standard system, which means an indicator system for CI resilience assessment must be established.

# 1. Introduction

On 28 February 2022, the Intergovernmental Panel on Climate
Change (IPCC) of the United Nations published a new report, Climate
Change 2022: Impacts, Adaptation and Vulnerability Pörtner et al. 2022,
aiming to assess the impacts of climate change, by looking at ecosys-
tems, biodiversity, and human communities at global and regional
levels. This report sets forth that most global territories are facing
increasing adverse impacts of climate change on infrastructures and
suggests integrated action for climate resilience to avoid natural di-
sasters, including planning and investment in urban infrastructure.
Meanwhile, the planet is also confronted with a growing number and

magnitude of technological and technical disasters worldwide. For
example, on 4 August 2020, an explosion in the main port in Beirut
because of poor management of stored products, resulted in damage to
many local infrastructures. In particular, three of Beirut's hospitals were
wrecked by the blast Dyer, 2020, and two others were damaged, which
added to the performance disruption of hospital infrastructures already
facing the COVID-19 crisis. Infrastructures, especially critical in-
frastructures (CIs), including buildings, roads, schools, and hospitals,
play a central role in modern city management when facing both tech-
nological and natural disasters. Therefore, in recent years, the popu-
larity of "CI resilience" has exploded in both academic and policy
discourses. The implemented actions for improving CIs based on

<footnote><sup>a</sup> Lab'urba, Université Gustave Eiffel, Champs-sur-Marne France</footnote>

<footnote><sup>b</sup> LATTS, UMR CNRS 8134 Université Gustave Eiffel/Ecole des Ponts ParisTech, Marne la Vallee, France</footnote>

<footnote><sup>c</sup> Laboratoire des Sciences des Risques (LSR), IMT mines Ales, Alès, France</footnote>

<footnote><sup>d</sup> Centre Génie Industriel, IMT Mines Albi, Université de Toulouse, Albi, France</footnote>

<footnote><sup>e</sup> CEREMA, Nantes, France</footnote>

<footnote>Abbreviations: CI(s), Critical Infrastructure(s); PrES, pre-event stage; DES, during event stage; PoES, post-event stage; NEPS, next event preparation stage.</footnote>

<footnote>* Corresponding author at: Lab'urba, Université Gustave Eiffel, Champs-sur-Marne, France.</footnote>

<footnote>E-mail address: zhuyu.yang@univ-eiffel.fr (Z. Yang).</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 1. Structure of this study.</figcaption></figure>

resilience assessment help reduce cities' vulnerability.

Resilience can be defined as the ability to absorb and adapt to a changing environment ISO, 2021. However, as the term “resilience” is broadly used in different contexts, it relates to various aspects, such as the capacity to manage with connected systems, to gracefully degrade and to improve the ability to cope with future risks Woods, 2019; Wied et al., 2020; Yang et al., (2022-a). The United Nations United Nations, 2016; United Nations, 2020 increasingly recognises the importance of city resilience in facing a crisis or disaster.

Recently, resilience assessment has become a key component of disaster management since it is a wildly accepted process that is used to understand the functionality and performance of a complex system exposed to disasters CORDIS-IMPROVER Project, 2016. “Resilience” is an abstract concept, and resilience assessment is a process for knowing its value or level by applying appropriate steps Tang, 2019. The approaches built for CI resilience assessment are frequently based on indicators (Hosseini, 2016; CORDIS-Smart Resilience Indicators for Smart Critical Infrastructures, 2018; Yang et al., (2022-a.). Indicator, being a less abstract concept than resilience, can be used to show positive or negative changes in resilience. Therefore, the indicator-based resilience assessment could help stakeholders to analyse CIs on a practical and situational basis and to make efficient decisions. The identification of indicators is considered key before assessing resilience. Indicators are based on basic data or accounts that are produced under strict quality assurance Eurostat, 2014. One other challenge of resilience assessment is therefore to transform data into actionable knowledge by the means of indicators Vogel and Canada, (1997); CORDIS-Smart Resilience Indicators for Smart Critical Infrastructures, 2018.

Until recently, there were not enough studies to review numerous existing scientific papers that focus on indicator-based assessment for CI resilience, and to analyse them based on indicators. The aim of this article is therefore: to select as much as possible the current scientific papers relevant to the CI resilience indicator; and to analyse the indicator-based resilience assessment in these papers.

## 2. Methodology and structure

The method used to develop this review consists of two principle steps that build two sections (sections 3 and 4) of this article, with which the reader could find the specific methods concern. In considering the aims of this study, these two steps are used respectively to define the significant elements to be investigated of this study, and to identify suitable scientific papers to be analysed (see Fig. 1). Both two steps are based on current extensive literature available relating to indicators-based assessment for CIs resilience.

As indicators-based resilience assessment is a broadly used method in science and practice, a large number of factors or elements in the current papers are worth noting. To define the valuable elements that this study should investigate in each reviewed paper, the step in section 3 is to give a brief overview of the key sectors involved in reference to highly recognised scientific papers. The selection of these highly recognised papers was based on the knowledge and interview of a dozen authors of this work, all from the relevant sectors. Section 4 shows the process of paper searches. Two methods are presented for identifying useful scientific papers: (1) the common search method through keywords on electronic scientific websites; and (2) the reference search method based on some high-level review papers, thereby addressing some of the weaknesses of Method (1). Section 5 presents the results in tables and graphs. In section 6, the study further analyses the results based on a discussion of the elements identified in section 3.

## 3. Key sectors and focused elements

The study of CIs resilience has developed rapidly in the last decades and indicator-based assessment has become an important part of it. The scientific team for this study considers the work to involve three sectors: “Critical Infrastructures”, “Resilience”, and “indicator-based assessment”. Moreover, the group has identified numerous highly recognised scientific papers in these three sectors after several exchanges. This section, taking into account these highly recognised papers, gives brief overviews of the common or significant aspects of these three sectors. As a result, based on this information, eight elements were elected to be investigated.

### 3.1. Critical infrastructures

There is no universally accepted standard for the classification of CIs. One of the reasons for this is that internationally there is no commonly accepted definition of CIs. Nevertheless, all definitions emphasise the contributing role of infrastructure to society or the potentially debilitating effect in the case of disruption CIPedia, 2020. For Brown et al. (2006), infrastructures that represent a significant public investment and where even minor disruptions can degrade the performance of systems and cause significant societal damage can be called critical infrastructures. Barroca et al. (2012) consider an infrastructure (i.e., a set of facilities providing services necessary for a city to function), as critical if its malfunctioning threatens the security, economy, lifestyle or public health of a city, region or even a state. For U.S Department of Homeland and Security U.S Department of Homeland and Security (2022), the infrastructure system, “whose incapacity or destruction would have a debilitation impact on the defence and economic security”, could be considered critical. The European Commission identifies CIs as those physical and information technology facilities, networks, services and assets that, if disrupted or destroyed, would have a serious impact on the health, safety, security or economic well-being of citizens or the effective functioning of governments in European Union countries EUR-Lex, 2010. The disruption of infrastructure would have a serious impact on the well-being, health, and safety of citizens and weaken the whole defence and economic organisation Rinaldi et al. 2001; Serre and Heinzle, 2018).

According to a review article by Moteff et al. (2003), it is difficult to
<table><caption>**Table 1**
Sectors of CIs.</caption><thead><tr><td>N°</td><td>Sectors of CIs</td></tr></thead><tbody><tr><td>1</td><td>Agriculture and Food</td></tr><tr><td>2</td><td>Airport</td></tr><tr><td>3</td><td>Banking</td></tr><tr><td>4</td><td>Chemical</td></tr><tr><td>5</td><td>Commercial Facilities</td></tr><tr><td>6</td><td>Communications</td></tr><tr><td>7</td><td>Critical Manufacturing</td></tr><tr><td>8</td><td>Dams, Emergency Services</td></tr><tr><td>9</td><td>Defence</td></tr><tr><td>10</td><td>Drinking Water and Water Treatment</td></tr><tr><td>11</td><td>Energy</td></tr><tr><td>12</td><td>Government Facilities</td></tr><tr><td>13</td><td>Industrial Base</td></tr><tr><td>14</td><td>Information Technology</td></tr><tr><td>15</td><td>Materials and Waste</td></tr><tr><td>16</td><td>National Monuments and Icons</td></tr><tr><td>17</td><td>Nuclear Reactors</td></tr><tr><td>18</td><td>Postal and Shipping</td></tr><tr><td>19</td><td>Public Health and Healthcare</td></tr><tr><td>20</td><td>Seaport/Harbour</td></tr><tr><td>21</td><td>Transportation Systems</td></tr></tbody></table>

Source: Proag (2014)

define what types of infrastructure are critical. The sectors of CIs, decided by each country, government or organisation, depends on their contexts and priorities. The Cybersecurity & Infrastructure security Agency of USA Cybersecurity & Infrastructure security Agency of USA (2020) updated 16 sectors of CIs, Chemical, Commercial facilities, Communications, Critical manufacturing, Dams, Defence industrial base, Emergency, Energy, Financial service, Food and agriculture, Government facilities, Healthcare and public health, Information technology, Nuclear reactors, Materials and waste, Transportation systems and Water and wastewater systems. CIs in Canada, called essential infrastructures (infrastructures essentielles), include Energy and Utilities, Information and Communication Technologies, Finance - Health, Food - Water, Transport - Security, Government and ManufacturingSecurité public Canada, (2022) European Commission (2005) divides CIs into 11 sectors: Energy, Information, Communication technologies, Water, Food, Health, Financial, Public & Legal Order and Safety, Civil administration, Transport, Chemical and nuclear industry, Space and research. This study suggests using 21 sectors defined by Proag (2014). Because,

according to his study, resilience is considered important in these CI sectors (see Table 1).

## 3.2. Resilience and CIs resilience

Similarly, the term “resilience” is multidisciplinary and difficult to define because of its very broad use. Etymologically, resilience comes from the Latin, resilio, resilire, which means a return and the ability to resume. Many resilience studies argue that Holling (1973) first introduced the concept for studying ecological science and defined “resilience” as a persistent ability to absorb change and disturbance and still maintain the same state variables. Over time, a series of interpretations of resilience has been presented. As the definition of the term resilience is not the focus of this article, some relevant review studies Manyena, 2006; Meerow et al., 2016; Hosseini et al., 2016; Martin-Breen and Anderies, 2011; Curt et al., 2018; Heinzle et al. 2022 are suggested here without further discussion. Overall, resilience has little orthodoxy in its conceptualization and application and becomes problematic when trying to measure it Cutter, 2016.

In the field of CIs, resilience has no broadly accepted definition either, but is highly related to engineering or socio-technical science Cutter et al., 2010; Smith and Stirling, 2010; Mottahedi et al., 2021). For infrastructures exposed to disasters and interdependent in modern society, the definition of resilience is extended with multiple aspects and perspectives. Among the various interpretations of resilience, this study favours the one that accords with the highly accepted perspective in the scientific community, focusing on engineering resilience and socio-technical dimension.

### 3.2.1. Properties and capabilities of CIs

The value or level of CI resilience can be described through the properties (like safety, sustainability, etc.) or capabilities of CIs from the narrower interpretation. Capabilities of a system could be capacities, characteristics, abilities, resources and knowledge Makadok, 2001; Renn, 2008; Hills, 2005; Eriksson and Juhl, 2012; Lindbom et al., 2015). Both property and capability are indeed used frequently for CI resilience. Leveson et al. (2017) focus on the safety of the socio-technical system for resilience assessment. Bruneau et al., (2003) emphasise the importance of five properties (capacity, robustness, redundancy, resourcefulness and rapidity) in engineered systems. Hollnagel (2014) mentions responding, monitoring, learning, anticipating ability,

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Stages of resilience scenario, source: Ouyang et al., 2012-a.; Francis and Bekera, 2014; Tendall et al., 2015; Yang et al., 2022-a.; CEREMA, 2020.</figcaption></figure>
sustainability and safety properties, and required knowledge for the
resilience of the built environment, including infrastructures. Barroca
and Serre Barroca and Serre, (2013) believe that cognitive competence
(i.e. having enough knowledge of CI and potential hazards) is one of the
most essential aspects for infrastructure resilience. Vulnerability and
fragility, highly related to resilience, are two characteristics of CIs that
describe the damaging effects of a hazard Proag, 2014; Argyroudis et al.,
(2019). Furthermore, from an organisational point of view, for CIs
exposed to natural hazards, resilience contains characteristics related to
mitigation, planning and prior disaster experience Cutter et al., 2010.
Ouyang et al. Ouyang et al., (2012-a.) highlight three capacities for
urban infrastructures: resistant, absorptive and restorative. Concerning
recovery ability, the availability of resources in the CIS system has been
usually mentioned (Turniquist and Vugrin, 2013; Ouyang and Wang,
2015).

3.2.2. *Temporal stages of resilience scenario*

Resilience could also be described through the change in perfor-
mance or function over time. Therefore the resilience scenario presents
with different temporal stages. Based on the existing theories of CI
resilience (Ouyang et al., 2012-a. Ouyang et al., 2012-a.; Francis and
Bekera, 2014; Tendall et al., 2015; CEREMA, 2020; Yang et al., 2022-a.,
this study divides the resilience scenario into four stages (see Fig. 2):

* pre-event stage (PrES): from the occurrence of a hazard to the beginning degradation of the function of CIs,

* during event stage (DES): from the beginning of the degradation to the maximum degradation of the function of CIs,

* post-event stage (PoES): from the maximum degradation of the function of CIs to the function returning to the level of the pre-event stage or recovering to an ideal state, structure or property (but still lower than the original state), and

* next event preparation stage (NEPS): from the function returning to the level of the pre-event stage to the occurrence of the next shock on CIs. This stage emphasises the ability of learning and improving from experience.

The NEPS in the former scenario is highly related to the PrES in the
next scenario. In addition, it is necessary to emphasise that the ability of
improvement may affect different following stages and scenarios. In
each scenario, performance should improve at least in one stage
compared to previous scenarios, thanks to the ability to learn and
improve from experience.

3.2.3. Effects of implemented actions

As Barroca and Serre (2013) argue, a resilient CI should have
different aspects of capabilities and involve actions to improve its ca-
pabilities. Potential action analysis helps identify decisions that should
be taken to reconcile objectives and constraints in the best possible
manner Alderson et al. (2015). Many researchers emphasise the benefit
of implemented actions in a single scenario. Zhang et al. (2015), Zhang
and Wang (2016), Ruas et al. (2018) recognise, for engineering-based
systems, the effects of preparedness action at PrES and adaptive
response actions at DES to recover efficiently and effectively. Zhang
et al. (2015), Sharifi and Yamagata (2016) mention the cost-benefit of
actions in discussing action efficiency. Madni and Jackson (2009)
consider resilience actions as immediate or short-term actions while
adaptation implies long-term learning. The latter highlights that the
benefit of implemented action beyond a single scenario corresponds
exactly to the improved capacity described above. Meanwhile, due to
the interaction of CIs and cities' components, the actions of one CI
potentially bring unexpected side effects on itself Leveson et al., 2017;
Costella et al., 2009; Hollnagel and Woods; 2017), even on the con-
nected urban components Robert and Hémond, 2012; Serre and Hein-
zlef, 2018). From the point of view of organisational management, the
positive or negative impact of an action is related to issues of efficiency

and safety, which determine the degree of satisfaction of the action
Atkinson et al., 2006; Tillement et al., 2009; Costella et al., 2009).
Therefore, the effects of implementing actions should also be a key
aspect during CI resilience assessment.

3.2.4. Interdependence

Recently, more and more CI resilience studies discuss the connec-
tions and interdependencies, which are primarily between CIs Rinaldi
et al., 2001; Reed et al., 2009; Ouyang and Wang, 2015; Serre, 2016),
sometime between CIs and non-technical components in human-environ-
ment Bambara et al., 2015; Yang et al., (2022-b).. Human-environ-
ment include physical, atmospheric, biological, social, economic and
politic components, conditions, and factors which impact the state,
condition, and quality of living conditions, employment, and health
Academies and of Sciences, Engineering, and Medicine. (1992). Nan
and Sansavini (2017) assess resilience by taking into account infra-
structure interdependence, which can correspond to one infrastructure
(internal interdependency) or more infrastructures (external interde-
pendency). Going beyond CI sectors, Markolf et al., 2018 rethink the
resilience of the interactions between infrastructures and socio-
ecological components. Similarly, Yang et al. (2022-a) believe that a
resilient infrastructure should have the ability to manage multiple
equilibriums with other urban systems. In modern society, due to the
interdependencies of all built environments, the damage of one single
CI, like physical disruption or dysfunction, frequently affects other
urban components. The resilience of interdependence between in-
frastructures requires each system to guarantee its functionality when
faced with the failure of other systems Serre and Heinzlef, 2018. This
trend of CI resilience comes from the awareness of the cascading effects
due to interconnected urban components. Increasing hazards require the
urban system to cope with potential cascading effects after negative
consequences on CIs. Furthermore, from a consequence-based approach,
the negative effects caused by used resources should also be taken into
account in the analysis of interdependencies Robert and Hémond, 2012.
This opinion is equally related to the side effects of implementing the
aforementioned actions.

Overall, CI resilience could be described through different proper-
ties, capabilities, stages and potential effects (positive, negative, side or
cascading) in considering implementing actions and interdependency.
The potential that occur after a disaster affects an infrastructure may
occur in the same infrastructure or its connected urban components. The
scope of urban components is broad, and could be categorised into socio-
economic systems, government organisation, nature and energy flow
systems, and technical-infrastructure systems Yang et al., 2022-b.. Ac-
cording to some historical events, the origin of catastrophic impacts of
CIs (disruption, construction, action, etc.) have already occurred on
socio-economic systems Chang, 2016; Tatano and Tsuchiya, 2008; Rose
and Wei, 2013), government organisation Marples, 2006; Rania et al.,
2019; Duffaut, 2013), nature and energy flow systems Gyau-Boakye,
2001; Buesseler et al. 2017; Aldeberky, 2007).

3.3. *Indicator-Based assessment*

Resilience assessment is a popular and common method in resilience
studies due to its effort to evaluate and characterise the characteristics of
a complex system Tang, 2019. Assessment is a process by which infor-
mation (could be data or an indicator) is obtained relative to some
known objective or goal Kizlik, 2012. The framework built for resilience
assessment, whether it is quantitative, qualitative or semi-quantitative,
is frequently based on indicators Hosseini et al., 2016; Cantelmi et al.,
2021. Quantitative approaches offer domain-agnostic measures to
quantify value across applications and structural-based modelling ap-
proaches that model domain-specific representations. Semi-quantitative
approaches provide a general numerical description of the classification,
without detailed formulae or models. Qualitative approaches are related
to approaches without a numerical descriptor and based on people's
## Indicator-based Resilience Assessment

<figure><img src="image_5.png" /><figcaption>Fig. 3. Factors and phases of “Indicator-based Resilience Assessment”.</figcaption></figure>

<table><caption>**Table 2**
Target elements to be identified in each selected paper.</caption><thead><tr><th>N°</th><th>Elements to be investigated</th><th>Description</th></tr></thead><tbody><tr><td>I</td><td>Capabilities or properties</td><td>Capabilities or properties in each reviewed paper, which could be characteristics, capacities, abilities, resources or knowledge described in relation to CI resilience. As capabilities are numerous and difficult to be listed, they were clarified during the lecture of selected papers.</td></tr><tr><td>II</td><td>Temporal stages</td><td>Temporal stages addressed in each paper, including pre-event stage (PrES), during-event stage (DES), post-event stage (PoES) and next event preparation stage (NEPS):<br>Whether these aspects are within its consideration during resilience assessment in the paper</td></tr><tr><td>III</td><td>Aspect of positive and negative effects of implemented action</td><td></td></tr><tr><td>IV</td><td>Aspect of cascading effects due to interdependence</td><td></td></tr><tr><td>V</td><td>Indicators</td><td>Indicators described, used and applied for indicator-based assessment in each paper</td></tr><tr><td>VI</td><td>Suitable CI sectors of indicators</td><td>Suitable CI sectors of identified indicators</td></tr><tr><td>VII</td><td>Resilience assessment method</td><td>Assessment approach in each paper, which could be quantitative, qualitative or semi-quantitative</td></tr><tr><td>VIII</td><td>Indicator assessment method</td><td></td></tr></tbody></table>

<table><caption>**Table 3**
Three review articles of reference search.</caption><thead><tr><th>Review article</th><th>Article</th></tr></thead><tbody><tr><td>Hosseini et al. (2016)</td><td>A review of definitions and measures of system resilience.</td></tr><tr><td>Meerow et al. (2016)</td><td>Defining urban resilience: A review. Landscape and urban planning</td></tr><tr><td>Curt and Tacnet (2018)</td><td>Resilience of critical infrastructures: Review and analysis of current approaches</td></tr></tbody></table>

<table><caption>**Table 4**
Synonyms or related vocabularies of keywords in three categories.</caption><thead><tr><th>Category</th><th>Keyword</th><th>Synonyms or related vocabularies</th></tr></thead><tbody><tr><td>1</td><td>Critical Infrastructure</td><td>all types of infrastructural systems or networks, technical systems, systems, engineered systems, engineering and industry (industrial)</td></tr><tr><td>2</td><td>Resilience</td><td>resilient, all capabilities and properties concerning CI resilience mentioned in section 3.</td></tr><tr><td>3</td><td>Indicator (for Assessment)</td><td>Index, criterion, criteria, metric, indices and some terms considered as general indicator (such as loss, damage, time, etc.)<br>Assess (assessing, assessed), Estimation (estimating, estimate, estimated), measurement (measure, measuring, measured), evaluation (evaluating, evaluated, evaluate), determining, simulation (simulating), quantification (quantifying, quantified, qualified), modelling</td></tr></tbody></table>

judgments and analysis, like surveyed experts or operators Hosseini et al., 2016; Cantelmi et al., 2021. Furthermore, qualitative approach includes also graphics or illustrations, which present the value or change of indicator and data without numerical calculations.

Resilience assessment is frequently based on indicators Cutter, 2016. To produce knowledge, which can be used to make decisions, and set

policies, requires information systems to transform data into information Peris-Mora et al., 2005; Bourgeois, 2014). Moreover, according to Eurostat Eurostat, 2014, indicator-based assessment consists of setting the expected evolution for the indicator by reference. Indicator-based resilience assessment could be therefore defined as an approach to knowing resilience value or level by observing indicators with a set reference. In the context of CIs resilience assessment, an indicator provides information to measure or assess the level of properties, functionality, or capabilities.

An indicator is measurable, associated with a criterion, intended to observe its evolution at defined intervals Yang et al., 2022-a. A criterion is a character or a sign, which is used to distinguish a thing, a concept or to make a judgment of appreciation. An indicator is objective information, whose assessment, like resilience assessment, could be quantitative, semi-quantitative or qualitative. For example, temperature could be an indicator assessed through objective quantitative methods, such as thermometer readings. Meanwhile, indicator assessment can be based on many subjective judgments, which we call qualitative assessment. For example, for an indicator, “the number of satisfied customers served”, the “number” is objective information, but the method of getting this information is based on the subjective judgements of customers. Semi-quantitative indicator assessment, for example, quantifies the number of satisfied customers, which is categorised by level as below 50 per cent and above 50 per cent; Temperatures are classified as below 20 degrees, between 20 and 40 degrees and above 40 degrees.

Moreover, the indicator measure needs reliable data Vogel and Canada, 1997; Cutter, 2016; CORDIS-Smart Resilience Indicators for Smart Critical Infrastructures, 2018). Data is a discrete fact, a raw element or the result of an observation, an acquisition, or a measurement, carried out by a natural or artificial instrument. Data are objective and don't have to function to evaluate or assess an object. For collective work to be done based on data, it is necessary to ensure a unit of format and a standard of measurement. The process that transforms numerous data into actionable indicators is called indicator assessment.

In conclusion, indicator-based resilience assessment for CIs refers to an approach including three factors, CI resilience, Usable indicator, Required data and two phases (see Fig. 3):

* resilience assessment: a process in which resilience values are obtained by usable indicators

* Indicator assessment: a process in which indicator values are obtained by reliable data

### 3.4. Focused elements

Each existing resilience assessment is based on appropriate steps designed by the creators or users themselves. In designing the steps, many elements of the assessment process need to be specified, such as criteria, indicators, assessment methods, data, indicator weights, etc. According to the current state presented above, this review work will focus on eight elements (see Table 2) of the assessment process in each of the selected papers. Since the assessment approach for CI resilience and indicator has been studied in several current research papers Hosseini et al., 2016; Cantelmi et al., 2021; Gasser et al., 2021; Sun et al. 2020), it is simply categorised into quantitative, qualitative and semi-quantitative.

### 4. Suitable papers and indicator identification

The online bibliographical databases are becoming an increasingly popular tool to screen scientific literature with key words Macedo-Rouet et al., 2012. The first paper search method is therefore based on an online database, Web of Science. Based on the aims of this study, “Critical infrastructure”, “Resilience” and “Indicator” are selected as key words. However, the limit of these keywords would miss some articles in the search scope, for example, a paper describing hospital building as
<figure><img src="image_6.png" /><figcaption>Fig. 4. Keyword search method.</figcaption></figure>

"hospital facility" Grimaz et al., 2021. This paper would not be selected through keyword search with "infrastructure". On the other hand, if "hospital facility" were also considered as a key word, the research result would include many articles not related to infrastructures. It illustrates how expanding the search to nearby terms can catch many irrelevant articles. This opinion has been mentioned in the study of Yang et al. (2022-a) in terms of the keyword "indicator". Therefore, a second approach, a reference search based on three recently popular review papers in resilience study Hosseini et al., 2016; Meerow et al., 2016; Curt and Tacnet, 2018), is used for the paper search work. Indeed, some existing tools for reference review, like PRISMA<fnref n="1" /> or CORTEX,<fnref n="2" /> are able to screen automatically by keywords. But again, for this study, keywords cannot be clearly defined, while the extension of the search to nearby terms would lead to select many unsuitable articles. Consequently, a reference search through manual selection is necessary and inevitable.

Paper searches in this study involve several manual steps, including selecting based on the title and abstract. During the operational process, three people perform these steps, but the whole research team decides the selection criteria. Each result has been discussed several times and approved by the research team. The keywords of suitable article selection are based on the identification of indicators because the indicator is the priority target of this article. Papers that do not contain indicators were excluded from our study.

## 4.1. Keyword Search Method

The Keyword Search Method in this study deploys and adjusts the methodology used in the review paper of Yang et al. (2022-a), which includes three main steps: Key words selection, Database screening and Suitability analysis. An electronic search with keywords "Critical infrastructure", "resilience" and "indicator" was applied on Web of Science on 25 January 2022. The search strings "critical infrastructure", "resilience" and "indicator" were set as "Topic" items, with the conjunction "and" and 128 papers were screened. The suitability analysis, based on the paper abstract, was applied to exclude the papers that match the keywords, but not the research interests. For example, Jeong and An (2016) discuss the usefulness and applicability of risk assessment from the point of view of the weight of key performance indicators (KPI), without description of concrete KPIs. Another example is from Wang et al. (2019) to assess energy infrastructure models from the resilience perspective, not the resilience of energy infrastructures. After excluding articles for which resources could not be found, 22 papers, which are considered to contain resilience indicators for CIs, were selected (see Fig. 4).

## 4.2. Reference Search Method

The reference search used in this study is based on three manual steps, 1) Review Articles Selection, 2) Selection through Title and 3) Suitability Analysis.

### 4.2.1. Review articles selection

The first consideration is that, due to the widespread use of the term resilience, there are already several review articles studying system resilience or CI resilience. These studies may not have the same purpose as this review study, but they make it possible to collect many references about resilience assessment. Three review articles are useful for this study (see table 3). Many articles were documented and discussed in these three review studies, so their references are browsed in detail and those that do not meet study interest are filtered out. Filtering consists of two steps: Selection through Title and Suitability Analysis.

### 4.2.2. Selection through Title

This step aims to observe whether the title of the reference contains the three keywords abovementioned for the Keyword Search Method and their synonyms or related vocabularies. All terms and vocabularies are divided in three categories (see table 4) corresponding to the three key sectors mentioned in section 3. The criterion is that the title must contain at least one element from each category. For example, if an article is entitled "Travel time of safety road networks", it is considered to correspond to criteria with its elements: "road networks" as category 1; "safety" as category 2; and "travel time" as category 3. We should point out, that the terms in this table are limitless. All types of "critical infrastructure", all capabilities or properties concerning "resilience" and all terms nearby "indicator" are recognised as relevant. Which terms can be considered relevant relies on the subjective judgement of the screener. Because of this, many words cannot be predicted before the reference search. Automatic searches are unreliable and manual searches are necessary.

### 4.2.3. Suitability analysis

Similar as the suitability analysis in the Keyword Search Method, this step aims to exclude the papers that don't match the research interests. The suitability analysis in the second method is more complex than the keyword search. In the keyword search, three applicable keywords guarantee every screened paper concerns "indicator", "resilience" and "critical infrastructure". The main task of suitable paper selection in the keyword search is checking whether the indicator mentioned is to assess CI resilience. However, for the second method, with a larger scope (see table 3), more information needs to be checked in reading papers' abstracts:

1) Whether a paper assesses the resilience, or relating capabilities and properties.

This task can be completed based on the definitions and examples listed in section 3.1. The goal is to observe whether papers assess CI capabilities or properties relating to resilience.

• 2) Whether the assessed system is critical infrastructure.

As we only have CI sectors (see table 1), not specific CI types, it is recommended to use the criteria in Moteff et al., (2003) to identify whether the target system is CI or not. Moteff et al., (2003) did not specify which infrastructures should be considered critical but argue "... for an infrastructure to be judged critical it must be vital to one or more broad national functions. That set of functions has expanded over time, beginning with national defence and economic security, to include public health and safety, and then national morale". As long as the system assessed in the paper meets this criterion, we can consider the system to be critical.

<footnote n="1">¹ https://www.prisma-statement.org/Default.aspx.</footnote>

<footnote n="2">² https://scirev.org/journal/cortex/.</footnote>
<figure><img src="image_7.png" /><figcaption>Fig. 5. Reference search method.</figcaption></figure>

• 3) Whether a paper applies the “indicator” function to other terms or terminologies, and whether they are used to assess CI resilience.

According to the study by Yang et al. (2022-a), some studies use other terms, which fit the definition of “indicator”, for assessment, such as indices, parameter and metrics. Like a second task, which terms or vocabularies should be considered as “indicator” could not be specified. This study focuses on the term or terminology fitting the definition of indicator (i.e., “a chosen piece of information, intended to observe its evolution of something”).

With the three mentioned pieces of information, we could finally identify the papers concerning indicator-based resilience assessment for CIs. Consequently, 20 papers were selected after the reference search based on the three review articles (see Fig. 5). Furthermore, in order to analyse more papers, this study applies a second time reference search method based on the 20 selected papers in the first reference search. 16 papers were selected after the second reference search (see Fig. 5. The details of suitability analysis are presented in Appendix 1 and 2 (see A.1 and A.2): “X” means that the article is suitable and selected for this study; “repeat” signifies the articles found through the second reference research were also found during the first reference research.

# 5. Results

All target elements in each paper relevant to this study are identified and listed in Fig. 6 and in three tables:

• Table 5. Results summary for the 22 papers selected through keyword search

• Table 6. Results summary for the 20 papers selected through first reference search

• Table 7. Results summary for the 16 papers selected through second reference search

From left to right, three tables illustrate paper numbers, references, capabilities or properties, temporal stages, relevance of interdependence, relevance of implementing action, identified indicators, suitable CI sectors of identified indicators, resilience assessment approach and indicator assessment approach. The same indicators used by several different documents appear in one column. Due to space limitations, the indicators in the original article have been simplified in this paper. This study only discusses the identified and exportable contents. Elements not discussed in papers are marked as Not Available (“NA”). For example, in the case of the first article in Table 5 from Kolowrocki and Soszyńska-Budny (2019), which does not clarify the resilience stages involved, this study prefers to note “NA” to avoid misinterpreting the author’s intentions.

“A” means that indicators are suitable for all CI sectors. We highlight

that suitable CI analysis of each indicator focuses on the indicator itself, not the CIs mentioned in papers. For example, LaLone et al. (2017) applies the indicator “human noise” on the power grid system, but this indicator is suitable for all types of CIs and noted as “A” meaning all sectors of CIs. In some cases, the suitable CI sectors of an identified indicator could be in several sectors, for example, “the days of CI disruption” Murdock et al., 2018 fitting all sectors and “number of passengers” Adjetey-Bahun et al., 2014 suitable for Transportation Systems, Seaport/Harbour and Airport. Moreover, this paper considers that the assessment of resilience is based on a whole indicators system that includes numerous indicators. Therefore, when an indicator is part of an indicator system, the identification of adapted CI sectors depends on the indicator system established by authors.

# 6. Results analysis

Before discussing the results, it is necessary to highlight that since resilience and CI are terms that don't have standard definitions, this review study is based on the aspects presented in section 3 and the research method designed by our research team. The discussion focusses on the popularity level in the field, rather than on the concrete quantity of each category in Fig. 6, as this study do not cover all of the existing papers, but is based on the 58 articles identified.

## 6.1. Capabilities or properties (I)

According to an analysis of the number of recurrences, among 125 “capabilities or properties” in total, the four most common words are performance (22 times), safety (11 times), vulnerability (8 times), recovery (8 times), accounting for more than 39 percent of all recurrences. The phenomenon that performance recurs most frequently can be explained by the fact that many articles have described resilience in terms of changes in performance over time. However, resilience is a multi-scale and complex concept Wang et al., 2019, which requires assessment relating to various perspectives and all temporal stages. An assessment aiming only at a part of capacities or stages could not assess CI resilience completely. The majority of investigated papers do not meet this requirement.

## 6.2. Temporal stages (II)

Fig. 6 shows that, except for the articles not concerning temporal stages, the majority of indicators aim at the During Event Stage (38 papers) and Post Event Stage (39 papers) of the resilience scenario. Only six articles focus on all stages of resilience scenarios. This proves that the analysis of preparedness and improvement stages of the resilience scenario is not enough. In particular, NEPS is not discussed in many reviewed papers. This study considers this stage highly relevant to the effectiveness of implementation actions. The lack of discussion of NEPS might be related to the insufficient focus on implementation actions.

## 6.3. Interdependence and implementing action (III and IV)

Moreover, positive or negative effects concerning “interdependencies” and “implemented actions” are not discussed enough in selected papers. Section 3 highlights that damage and implemented actions of one CI could cause effects over time. Positive effects refer to the Effectiveness of actions, which could be reflected in various ways, such as the increase of functionality, the change of functional reduction in speed and degree, and the speed of functional recovery. Negative effects signify: 1) the damage caused by implemented actions to target CI and its connected components in human-environment; and 2) the cascading damage to target CI and connected urban components due to interdependencies. Both positive and negative effects are likely to occur over the long term. Resilience indicators should go beyond the boundaries of one shock event or one CI and help create an ongoing resilience
<figure><img src="image_8.png" /><figcaption>Fig 6. Results summary for all selected papers.</figcaption></figure>
<table><caption>**Table 5**
Results summary for the papers selected through keyword search.</caption><thead><tr><th rowspan="2">N°</th><th rowspan="2">References</th><th rowspan="2">I<br>Capabilities<br>or properties</th><th rowspan="2">II<br>Temporal<br>stages</th><th rowspan="2">III<br>Implementing<br>action</th><th rowspan="2">IV<br>Interdependence</th><th rowspan="2">V<br>Indicators</th><th rowspan="2">VI<br>CI<br>Sectors</th><th colspan="2">VII<br>Assessment methods</th></tr><tr><th>Resilience</th><th>Indicator</th></tr></thead><tbody><tr><td>1</td><td>Kołowrocki and<br>Soszyńska-budny,<br>(2019)</td><td>Safety</td><td>NA</td><td>No</td><td>No</td><td>safety function; risk<br>function; fragility curve;<br>critical infrastructure</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>2</td><td>Kołowrocki, 2020</td><td>Safety</td><td>NA</td><td>No</td><td>No</td><td>unconditional lifetime up<br>to exceeding critical</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>3</td><td>Kołowrocki et al.,<br>2019-a</td><td>Safety</td><td>NA</td><td>No</td><td>No</td><td>safety state; the standard<br>deviation of the critical<br>infrastructure lifetime in<br>the safety state not worse<br>than the critical state; the<br>moment of exceeding the<br>acceptable value of the<br>critical infrastructure risk<br>function level; the<br>intensities of degradation<br>of the critical<br>infrastructure; the<br>intensities of critical<br>infrastructure departure<br>from the safety state<br>subset; the limit values of<br>transient probabilities of<br>the operation process at<br>the particular operation<br>states; intensities of<br>ageing at the operation<br>states</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>4</td><td>Kołowrocki et al.<br>(2018)</td><td>Safety</td><td>NA</td><td>No</td><td>No</td><td>the unconditional safety<br>function; risk function;<br>fragility curve;</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>5</td><td>Kołowrocki et al.,<br>2019-b</td><td>Safety</td><td>NA</td><td>No</td><td>No</td><td>unconditional CI lifetime<br>in the subset of safety<br>state, times of the CI<br>staying at particular states<br>of safety; CI lifetime<br>standard deviation of the<br>CI staying in the subset of<br>safety states, the moment<br>of exceeding the<br>acceptable value of the<br>critical infrastructure risk<br>function level; the<br>coefficients of the climate-<br>weather change influence<br>on CI ageing intensities;<br>process influence on CI;<br>the limit value of the<br>process influence on CI;<br>the mean value of the<br>process influence on CI;<br>intensities of ageing at the<br>operation states</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>6</td><td>Hromada and<br>Lukas, (2012)</td><td>Vulnerability,<br>Coefficients (of the<br>relationships and<br>interdependencies of<br>risks)<br><br>Countermeasures<br>(ability to recover):<br>Detection, Delay,<br>Response, quality</td><td>PrES<br>DES<br>PoES</td><td>Yes</td><td>No</td><td>activity coefficient;<br>passivity coefficient (sub-<br>indicator: total number of<br>risks, possibility of risk,<br>risk importance<br>correlation, risk<br>coefficients)<br>the time needed to verify<br>alarm information and<br>probability of successful<br>communication; breaking<br>resistance; the time<br>needed for the guards to<br>transfer from A to B; sum<br>of positive answers<br>human noise</td><td>A</td><td>Quantitative</td><td>NA</td></tr><tr><td>7</td><td>LaLone et al.<br>(2017)</td><td>Recovery efforts</td><td>DES<br>PoES</td><td>No</td><td>No</td><td></td><td>A</td><td>Quantitative</td><td>Qualitative</td></tr><tr><td>8</td><td>Bialas (2016)</td><td>Effectiveness,<br>Efficiency</td><td>NA</td><td>Yes</td><td>No</td><td>A large number in four<br>aspects: security, safety,<br>reliability, resilience,</td><td>A</td><td>Qualitative</td><td>NA</td></tr></tbody></table>

(continued on next page)
<table><caption>**Table 5 (continued)**</caption><thead><tr><th rowspan="2">N°</th><th rowspan="2">References</th><th rowspan="2">I<br>Capabilities<br>or properties</th><th rowspan="2">II<br>Temporal<br>stages</th><th rowspan="2">III<br>Implementing<br>action</th><th rowspan="2">IV<br>Interdependence</th><th rowspan="2">V<br>Indicators</th><th rowspan="2">VI<br>CI<br>Sectors</th><th colspan="2">VII<br>Assessment methods</th></tr><tr><th>Resilience</th><th>Indicator</th></tr></thead><tbody><tr><td>9</td><td>Liu et al. (2021)</td><td>Mitigation, Recovery,<br>Performance</td><td>PrES<br>DES<br>PoES</td><td>No</td><td>Yes</td><td>technical and<br>management issues; CI<br>specific phenomena; cost<br>and benefit parameters<br>dealing with the entire CI<br>protection system;<br>Incident statistics (for<br>details see the list in the<br>reference)</td><td>A</td><td>NA</td><td>Quantitative</td></tr><tr><td>10</td><td>Kamissoko et al.<br>(2020)</td><td>Sensitivity<br>Functionality</td><td>PrES<br>DES<br>PoES</td><td>No</td><td>Yes</td><td>resilience by mitigation<br>and recovery (sub-<br>indictor: performance<br>function, time,<br>complement of<br>insufficiency function)<br>failure magnitudes and<br>recovery rates<br>the number of trucks in<br>circulation (sub-indictor:<br>the performance, the<br>safety and the number of<br>circulating trucks)<br>the electricity used, the<br>quantity of required<br>electricity<br>the number of good<br>functioning periods; the<br>number of bad<br>functioning periods when<br>the value is greater than<br>the normal maximum; the<br>number of bad<br>functioning periods when<br>the value is lower than the<br>normal minimum; the<br>duration of all bad<br>functioning periods;</td><td>21</td><td>Quantitative</td><td>NA</td></tr><tr><td>11</td><td>Roe and Schulman<br>(2012)</td><td>Performance<br>reliability, Resource-<br>benefit</td><td>PrES</td><td>Yes</td><td>No</td><td>the quantity of electricity<br>produced, the number of<br>incoming trucks<br>variables include daily or<br>hourly load forecast<br>errors, hardware<br>malfunctions, grid<br>congestion and software<br>glitches; the impact<br>imposed by the input<br>conditions of unscheduled<br>generator outages on the<br>grid; line mitigations<br>disruption days (sub-<br>indictor: the number of<br>people affected by the<br>disruption in CI services<br>and the days of<br>disruption)</td><td>11</td><td>Qualitative</td><td>Quantitative</td></tr><tr><td>12</td><td>Murdock et al.<br>(2018)</td><td>Redundancy,<br>Flexibility,<br>Robustness</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>failure duration; failure<br>magnitude; severity; the<br>global surplus hydraulic<br>power above the<br>minimum required nodal<br>water head; Capital costs;<br>Operational cost<br>capacity; damage level;<br>probability of structural<br>collapse; liability; direct<br>risk associated with the<br>cost of local failure due to<br>exposure to a hazard,<br>indirect risk associated<br>with the cost of collapse<br>given local damage;</td><td>10</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>13</td><td>Zhan et al. (2020)</td><td>Reliability,<br>Performance</td><td>DES<br>PoES</td><td>No</td><td>No</td><td></td><td></td><td></td><td></td></tr><tr><td>14</td><td>Ghosn et al.<br>(2016)</td><td>Redundancy, Safety<br>Fragility,<br>Vulnerability,<br>Robustness,<br>Functionality</td><td>PrES<br>DES<br>PoES<br>NEPS</td><td>No</td><td>Yes</td><td></td><td></td><td></td><td></td></tr></tbody></table>

(continued on next page)
<table><caption>Table 5 (continued)</caption><thead><tr><th rowspan="2">N°</th><th rowspan="2">References</th><th rowspan="2">I<br>Capabilities<br>or properties</th><th rowspan="2">II<br>Temporal<br>stages</th><th rowspan="2">III<br>Implementing<br>action</th><th rowspan="2">IV<br>Interdependence</th><th rowspan="2">V<br>Indicators</th><th rowspan="2">VI<br>CI<br>Sectors</th><th colspan="2">VII<br>Assessment methods</th></tr><tr><th>Resilience</th><th>Indicator</th></tr></thead><tbody><tr><td></td><td></td><td>Performance</td><td></td><td></td><td></td><td>connectivity loss;<br>efficiency; time; statistical<br>independence; Service<br>flow reduction network;<br>topology<br>Interruption duration;<br>average customer<br>interruption duration;<br>average system<br>interruption frequency<br>all nodes are connected to<br>the source nodes<br>total number of highway<br>sections open; total length<br>of highway open; total<br>weighted "connected"<br>length of highway open;<br>travel time; travel<br>distance; user cost</td><td>21</td><td></td><td></td></tr><tr><td>15</td><td>Upadhyaya et al.<br>(2018)</td><td>Functionality</td><td>PrES<br>DES<br>PoES<br>NEPS</td><td>Yes</td><td>No</td><td>dry weather/wet weather<br>flow; monitoring of the<br>demographic pattern;<br>Change in impervious<br>area; etc. (for details see<br>the list in references)<br>assessment of potential<br>damage; assessment of<br>reconstruction need;<br>Recovery plan; etc. (for<br>details see the list in<br>references)<br>Energy used to provide<br>services/year; excess<br>toxicity in incoming<br>water; etc. (for details see<br>the list in references)</td><td>10</td><td>Quantitative</td><td>Semi-<br>quantitative</td></tr><tr><td></td><td></td><td>Resiliency</td><td></td><td></td><td></td><td>accessibility; Road<br>density; Road width; Road<br>lanes; Link (road, track,<br>etc.) condition; etc. (for<br>details see the list in<br>references)</td><td></td><td></td><td></td></tr><tr><td></td><td></td><td>Sustainability</td><td></td><td></td><td></td><td>the number of equivalent<br>households without<br>service loss function<br>the tank water level loss<br>function<br>water quality<br>levels of floods; the<br>damaged area; the<br>number of affected links;<br>the number of affected<br>nodes and the number of<br>affected cars</td><td></td><td></td><td></td></tr><tr><td>16</td><td>Kammouh et al.<br>(2018)</td><td>NA</td><td>NA</td><td>No</td><td>No</td><td>the height of the<br>nitrification-oxidation<br>process tank; The power<br>delivered to the pump;<br>The outlet mass flow rate<br>the technical "up time"<br>and stability of<br>decentralised<br>communication<br>equipment; The number<br>of unwanted incidents<br>related to communication<br>failures; The regularity of<br>railway operations and<br>delays due to incidents.<br>score of culture rating<br>based on the survey<br>extreme temperatures<br>(Increase in frequency of</td><td>21</td><td>NA</td><td>Quantitative</td></tr><tr><td>17</td><td>Cimellaro et al.<br>(2016)</td><td>Performance function</td><td>PrES<br>DES<br>PoES<br>NEPS</td><td>No</td><td>No</td><td></td><td>10</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>18</td><td>Tachaudomdach<br>et al. (2021)</td><td>Robustness</td><td>PrES</td><td>No</td><td>No</td><td></td><td>21</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>19</td><td>Murino, et al.<br>(2019)</td><td>Performance</td><td>DES<br>PoES</td><td>No</td><td>No</td><td></td><td>10</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>20</td><td>Johnsen and Veen<br>(2013)</td><td>Safety, Security<br>culture</td><td>DES<br>PoES<br>NEPS</td><td>No</td><td>No</td><td></td><td>A</td><td>Qualitative</td><td>NA</td></tr><tr><td>21</td><td>De Vivo et al.<br>(2022)</td><td>Vulnerability,</td><td>NA</td><td>No</td><td>No</td><td></td><td>2</td><td>NA</td><td>NA</td></tr></tbody></table>

(continued on next page)
<table><caption>Table 5 (continued)</caption><thead><tr><th rowspan="2">N°</th><th rowspan="2">References</th><th rowspan="2">I<br>Capabilities<br>or properties</th><th rowspan="2">II<br>Temporal<br>stages</th><th rowspan="2">III<br>Implementing<br>action</th><th rowspan="2">IV<br>Interdependence</th><th rowspan="2">V<br>Indicators</th><th rowspan="2">VI<br>CI<br>Sectors</th><th colspan="2">VII<br>Assessment methods</th><th rowspan="2">VIII<br>Indicator</th></tr><tr><th>Resilience</th><th>Indicator</th></tr></thead><tbody><tr><td></td><td></td><td>Sensitivity,<br>Adaptive capacity</td><td></td><td></td><td></td><td>extreme temperatures;<br>Increase in intensity of<br>extreme temperatures),<br>extreme rainfall (Increase<br>in rainfall frequency;<br>Increase in rainfall<br>intensity); sea level rise;<br>Soil sealing; Air traffic,<br>etc. (for details see the list<br>in references)</td><td></td><td></td><td></td><td></td></tr><tr><td>22</td><td>Pietrucha-Urbanik<br>et al. (2020)</td><td>reliability</td><td>DES</td><td>No</td><td>No</td><td>disruption frequency;<br>network downtime;<br>network repair time and<br>network back-to-service<br>time; unavoidable annual<br>real water losses and the<br>infrastructure leakage</td><td>10</td><td>Qualitative</td><td>Quantitative</td><td></td></tr></tbody></table>

assessment for a CI, as well as for several interdependent urban components. Therefore, both the aspects concerning CIs' interdependence and effects caused by implemented actions should be taken into account for resilience assessment. However, among these selected papers, only 6 articles mention interdependence, while 13 papers relate to the implementing actions concerning cost, resources, effectiveness and safety of actions.

## 6.4. Indicators (V)

The identification of indicators is a complex process. Firstly, only 30 of the 58 articles use the term “indicator” and specifically explain what they are. Some papers apply the function of “indicator” to its synonyms or related vocabularies, like parameter Hromada and Lukas, 2012; Liu et al.,2021), variable Roe and Schulman, 2012; Murino, et al., 2019; Baroud et al., 2014b; Morlok and Chang, 2004, metric (Ghosn et al., 2016; Shafieezadeh and Ivey Burden, 2014; Ouyang and Wang, 2015; Ouyang and Duenas-Osorio, 2014; Vugrin et al., 2010; Henry and Ramirez-Marquez, 2012; Pant et al., 2014; Cavalieri et al., 2012; Ayyub, 2013), index Ghosn et al., 2020; Cimellaro et al. 2016; Ayyub, 2014), etc. These vocabularies are closely associated with indicator-based assessment, but have various definitions in different contexts. If they have the function of “indicator” in reviewed papers, they are presented in Table 5, 6 and 7. On the other hand, some mentioned indicators in reviewed papers are not for assessing resilience. For example, two papers consider “resilience” as an indicator for assessing risk-cost trade-offs in water resources system Li and Lence, 2017 and for assessing recovery capability in Intermodal Freight Transport Chen and Miller-Hooks, 2012. In these cases, we chose the information assessing “indicator resilience” as indicators.

Sometimes the 'indicators' are not presented in an obvious way. For example, Johnsen and Veen (2013) calculate the scores of three issues relating to safety culture, based on a questionnaire-based survey, and the number of scores could be considered as an indicator. Vugrin ED et al. (2010) use a number of metrics to assess performance, whose values in different periods change into the values of “recovery path” and “recovery effort”. Then interviewers present these results to experts, who use them to assess the resilience of CIs. Finally, CI resilience is classified as high, medium or low-level. Therefore, what could be considered as a resilience indicator in the whole assessment process of Vugrin ED et al., (2010) is the “results of interviews”, which are an assessment of performance, recovery path and recovery effort. Some papers Faturechi and Miller-Hooks, 2014; Reed et al., 2009; Ip and Wang, 2011; Sun et al., 2006; Omer et al., 2015; Li and Lence, 2017; Miller-Hooks et al., 2012; Winkler et al., 2010; Chang, 2003) provide

mathematical formulas to calculate resilience or elastic capacity, and this study records the variables needed for the calculation as an indicator. In some other papers Carvalho et al., 2008; Ouyang and Duenas-Osorio, 2012-b.; Gilbo, 1993, the indicators were obtained from the authors' analysis.

The results also show that the term “indicator” is sometime misused. Adams et al. (2012) apply “redundancies,” “recovery activities” and “travel speed” as indicators, but the former two are not objective enough to be an indicator. This problem occurs in many other papers, such as Shirali et al. (2012), Shirali et al. (2013), LaLone et al. (2017), Liu et al. (2021), Shafieezadeh and Ivey Burden, (2014), etc. The subjective terms like availability, resilience, mitigation and recovery are related to ideal expectations of stakeholders, which cannot be an indicator since it must be objective. Such as the “availability of cranes and berths” indicator in Liu et al. (2021), whether it should correspond to crane volumes or quantity of vehicle berths, since the value that is considered to represent the expected performance of that road infrastructure depends on the subjective judgment of stakeholders. Human judgement for expected performance is based on their means, objectives and results Bescos et al., 1997. Therefore, the non-objective terms are not indicators, but called criteria that are used to distinguish a thing or a concept, and to make a judgment of appreciation.

## 6.5. Suitable CI sectors (VI)

36 papers present the indicators that could be applied to all types of CIs. These indicators are relatively general in comparing the indicators only suitable for specific sectors. Moreover, this study identifies more indicators specific to two CI sectors: transportation systems, Drinking Water and Water Treatment. Specific indicators for 10 sectors are not present in the results: Banking, Commercial Facilities, Critical Manufacturing, Defence, Government Facilities, Industrial Base, Materials and Waste, National Monuments and Icons, and Nuclear Reactors. This result does not indicate a low level of interest in these sectors. For example, although we did not identify indicators applicable only to Industrial Base sectors, the studies on industrial plants could be found in some review papers, like Shirali et al. (2012), and Shirali et al. (2013). It can only be said that the indicators they use are also applicable to other CI sectors. A hypothesis is proposed, but it needs to be proven: more specific indicators are needed to assess the infrastructure in several sectors, like transport and water systems, where such needs do not exist in other sectors. Meanwhile, when reading the paper, we found that some CI sectors are indeed less discussed for resilience assessment, especially National Monuments and Icons, Materials and Waste, Banking, and Government Facilities.
<table><caption>**Table 6**
Results summary for the paper selected through first reference search.</caption><thead><tr><th rowspan="2">N°</th><th rowspan="2">References</th><th rowspan="2">I<br>Capabilities<br>or properties</th><th rowspan="2">II<br>Temporal<br>stages</th><th rowspan="2">III<br>Interdependence</th><th rowspan="2">IV<br>Implementing<br>action</th><th rowspan="2">V<br>Indicators</th><th rowspan="2">VI<br>CI Sectors</th><th colspan="2">VII<br>Assessment methods</th></tr><tr><th>Resilience</th><th>Indicator</th></tr></thead><tbody><tr><td>1</td><td>Shafieezadeh and<br>Ivey Burden,<br>(2014)</td><td>available capacity<br>Performance</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>availability of cranes and berths<br>time required and service provided<br>(sub-indictor: the number of berthed<br>ships, the delay time (i.e., the time in<br>between ship arrival and docking), the<br>number of displaced ships (i.e., the<br>number of ships within the arrival<br>stream that cannot be accommodated at<br>the port on a given day due to large<br>delay times), and the number of TEUs<br>handled by the seaport.)</td><td>20</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>2</td><td>Ouyang and<br>Wang (2015)</td><td>Restoration,<br>Performance,<br>Fragility (of<br>components)</td><td>DES<br>PoES</td><td>yes</td><td>No</td><td>event occurrence number; total number<br>of event occurrences; occurrence time<br>of the event; impact area; expected<br>impact area under the hazards<br>accounting for all possible intensities;<br>occurrence rate of the hazards per year</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>3</td><td>Ouyang and<br>Duenas-Osorio,<br>(2014)</td><td>Restoration,<br>Performance,<br>Fragility (of<br>components)</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>event occurrence number; total number<br>of event occurrences; occurrence time<br>of the event; impact area; expected<br>impact area under the hazards<br>accounting for all possible intensities;<br>occurrence rate of the hazards per year</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>4</td><td>Vugrin et al.<br>(2010)</td><td>Absorptive<br>capacity, Adaptive<br>capacity,<br>Restorative<br>capacity<br>Performance</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>the profitability of power companies;<br>the percentage of customers with<br>working electric power<br>rates of and population exposure to food<br>contamination; average consumer price<br>of food<br>shipments to critical chemical-based<br>commodities<br>lives saved; average response time<br>barrels of refined petroleum product<br>transported to the Midwest; price of<br>domestic refined products; profitability<br>of energy companies<br>number and efficacies of cyber attacks<br>rates of morbidity and mortality; cost<br>per vaccine given<br>average speed and cost of shipments;<br>number of disrupted shipments<br>number of dropped telephone calls</td><td>11<br>1<br>4.18<br>8<br>11<br>14<br>19<br>20<br>6</td><td>Qualitative</td><td>Qualitative</td></tr><tr><td>5</td><td>Shirali et al.<br>(2012)</td><td>Safety culture</td><td>PrES<br>DES<br>PoES<br>NEPS</td><td>No</td><td>No</td><td>schedule delays; (frequency of) safety<br>committees; meeting effectiveness;<br>safety education; worker's involvement;<br>competence; safety training</td><td>A</td><td>Qualitative</td><td>Qualitative</td></tr><tr><td>6</td><td>Shirali et al.<br>(2013)</td><td>Safety culture</td><td>PrES<br>DES<br>PoES<br>NEPS</td><td>No</td><td>No</td><td>top management; commitment;<br>learning culture; awareness;<br>preparedness; and flexibility.</td><td>A</td><td>NA</td><td>Qualitative</td></tr><tr><td>7</td><td>Adams et al.<br>(2012)</td><td>Reduction,<br>Recovery,<br>Performance</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>redundancies; recovery activities; travel<br>speed (sub-indictor: the severity of<br>damage on travel speed and truck<br>counts, reduction period time, recovery<br>period time)<br>function; time; cost of resilience</td><td>21</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>8</td><td>Henry and<br>Emmanuel<br>Ramirez-<br>Marquez, (2012)</td><td>NA</td><td>DES<br>PoES</td><td>No</td><td>No</td><td></td><td>A</td><td>Quantitative</td><td>NA</td></tr><tr><td>9</td><td>Pant et al. (2014)</td><td>Vulnerability,<br>Recoverability</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>time to Total System Restoration; Time<br>to Full System Service Resilience; and<br>Time to 100 % Resilience.</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>10</td><td>Baroud et al.,<br>(2014b)</td><td>Reliability,<br>Vulnerability,<br>Recoverability</td><td>PrES<br>DES<br>PoES</td><td>No</td><td>No</td><td>time to total system restoration; time to<br>full system service resilience; and time<br>to a specific % resilience; the number of<br>links recovered</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr></tbody></table>

(continued on next page)
<table><caption>**Table 6 (continued)**</caption><thead><tr><th rowspan="2">N°</th><th rowspan="2">References</th><th rowspan="2">I<br>Capabilities<br>or properties</th><th rowspan="2">II<br>Temporal<br>stages</th><th rowspan="2">III<br>Interdependence</th><th rowspan="2">IV<br>Implementing<br>action</th><th rowspan="2">V<br>Indicators</th><th rowspan="2">VI<br>CI Sectors</th><th colspan="2">VII<br>Assessment methods</th></tr><tr><th>Resilience</th><th>Indicator</th></tr></thead><tbody><tr><td>11</td><td>Chen and Miller-Hooks (2012)</td><td>Redundancy, Recovery</td><td>DES<br>PoES</td><td>No</td><td>yes</td><td>post-disruption expected fraction of demand (time, cost, resources); redundancies; recovery activities</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>12</td><td>Ayyub (2014)</td><td>Performance, Quality</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>time of incident or disturbance occurrence; time to full recovery; The probability of cost; standard normal cumulative distribution function space availability</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td>throughput traffic</td><td>A<br>2.18.20.21</td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td>water production capacity</td><td>10</td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td>water available for consumption</td><td>10</td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td>power delivered</td><td>11</td><td></td><td></td></tr><tr><td>13</td><td>Hashimoto et al., (1982)</td><td>Reliability, Vulnerability, Performance</td><td>PrES<br>DES<br>PoES</td><td>No</td><td>No</td><td>average level and average squared of water deviation</td><td>10</td><td>Quantitative</td><td>NA</td></tr><tr><td>14</td><td>Franchin and Cavalieri (2015)</td><td>Efficiency</td><td>PoES</td><td>No</td><td>No</td><td>function</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>15</td><td>Faturechi et al. (2014)</td><td>NA</td><td>PoES</td><td>No</td><td>yes</td><td>post-repair capacity rates for take-offs and landings</td><td>2</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>16</td><td>Faturechi and Miller-Hooks (2014)</td><td>NA</td><td>PrES<br>DES<br>PoES</td><td>No</td><td>No</td><td>travel time</td><td>21</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>17</td><td>Adjetey-Bahun et al.(2014)</td><td>Performance</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>the number of passengers that reach their destination; the total delay of passengers after a serious perturbation.</td><td>21</td><td>Qualitative</td><td>Quantitative</td></tr><tr><td>18</td><td>Enjalbert et al. (2011)</td><td>Performance, Safety</td><td>PrES<br>DES<br>PoES</td><td>No</td><td>Yes</td><td>the evolution of performances (sub-indicator: time period during which the performance improvement occurs or remains; the consequences of human actions in order to compare performance levels between two dates; the effect of disturbance on the system sinkhole-Subsidence cone; Differential movement; Concrete quality; Visual state of drain outlet; Visual state of drain collector; etc. (for details see the list in references))</td><td>A</td><td>Quantitative</td><td>NA</td></tr><tr><td>19</td><td>Curt et al., (2010)</td><td>Performance</td><td>NA</td><td>No</td><td>No</td><td>clustering coefficient; connection points</td><td>8</td><td>Quantitative, Semi-quantitative</td><td>Semi-quantitative</td></tr><tr><td>20</td><td>Lhomme et al. (2013)</td><td>Redundancy</td><td>PoES</td><td>No</td><td>No</td><td></td><td>A</td><td>NA</td><td>NA</td></tr></tbody></table>

## 6.6. Resilience and indicator assessment (VII and VIII)

Most reviewed papers use a quantitative assessment approach, both in the indicator and resilience assessment phases. In the “Resilience assessment” phase, the quantitative approach is 44 times higher than the semi-qualitative method, and 6.3 times higher than the qualitative approach. In the “Indicator assessment” phase, these values were 19.5 and 9.75 respectively. Quantitative assessment is frequently related to modelling, simulation and general mathematic equations, while Qualitative assessment is generally based on questionnaires Vugrin et al., 2010; Shirali et al., 2012; Shirali et al. 2013) and graphical comparisons Pietrucha-Urbanik et al., 2020; Adjetey-Bahun et al., 2014; Carvalho et al., 2008). For example, LaLone et al. (2017) draw the change of social indicators based on the data of Tweets per hour that mention keywords and Customers affected by Power Outages during hurricanes.

The assessment for the same capabilities or properties may have different approaches. One example comes from the reliability assessment of the water supply network, related to three articles. Zhan et al. (2020) and Li and Lence (2007) apply the quantitative assessment method in both resilience and indicator assessment phases. Pietrucha-Urbanik et al. (2020) use quantitative methods to calculate the values of two indicators (failure rates and water losses), and then compare graphically the change of indicators to qualitatively assess the reliability of the water supply network during different periods. Besides, for the same capabilities or properties, applied indicators are also different. For example, both relating to CI quality, Reed et al. (2009) present a greater number of indicators, including “inoperability”, “wind speed for the

hurricane”, “capacity”, “time in days post-event”, etc., while Hromada and Lukas (2012) apply indicators including “delay of breaking resistance”, “information and probability of successful communication”, “total number of risk”, etc. The selection of indicators that are more accurate or practical should be an invaluable issue.

From a psychological point of view, assessing an object is a process that involves judgment and decision-making, which apply certain criteria, principles or standards to form an assessment Sun et al., 2019. For example, Hollnagel (2015) classifies “Potential of Resilience Performance” into five levels: Excellent, Satisfactory, Acceptable, Unacceptable and Deficient. The “Potential of Resilience Performance” is subjective, and can be considered as a criterion that can be used to make judgements about system resilience. However, “criterion” and “criteria” for resilience indicators are discussed only in 12 papers Liu et al., 2021; Upadhyaya et al., 2018; Ghosn et al, 2016; Hashimoto et al., 1982; Kamissoko et al., 2020; Tachaudomdach et al., 2021; Adams et al., 2021; Enjalbert et al., 2011; Curt et al., 2010; Cavalieri et al., 2012; Gilbo, 1993; and Chang, 2003. What’s more, in these papers the criteria or indicators may be somewhat confusing, due to the misuse of terms or unclear expressions. For example, Kamissoko et al. (2020) present 6 criteria for three interdependent CIs: “the Quantity of Produced Electricity”, “the Number of Incoming Trucks”, “the Quantity of Required Electricity”, “the Performance”, “the Safety” and “the Number of Circulating Trucks”. But apart from Performance and Safety, all the other elements could be supposed to be objective information and should be called indicators. Conversely, many expressions are not objective enough to be an indicator, but are used as such.
<table><caption>**Table 7**

Results summary for the paper selected through second reference search.</caption><thead><tr><th rowspan="2">N°</th><th rowspan="2">References</th><th rowspan="2">I<br>Capabilities<br>or properties</th><th rowspan="2">II<br>Temporal<br>stages</th><th rowspan="2">III<br>Interdependence</th><th rowspan="2">IV<br>Implementing<br>action</th><th rowspan="2">V<br>Indicators</th><th rowspan="2">VI<br>CI<br>Sectors</th><th colspan="2">VII<br>Assessment methods</th></tr><tr><th>Resilience</th><th>Indicator</th></tr></thead><tbody><tr><td>1</td><td>Serre (2018)</td><td>Resistance,<br>Absorptive,<br>Recovery</td><td>PrES<br>DES<br>PoES</td><td>No</td><td>yes</td><td>contribution of the urban design features in DS3 (spatial decision support system) model</td><td>A</td><td>Qualitative</td><td>NA</td></tr><tr><td>2</td><td>Fisher et al.<br>(2010)</td><td>15 factors in two levels (see level 1 and 2 in references)</td><td>PrES<br>DES<br>PoES<br>NEPS</td><td>No</td><td>yes</td><td>Answers (yes or no) for level 5 in this study, for example: if “Multiple service lines” are in different geographic locations. (for details see the list in references)</td><td>A</td><td>Quantitative</td><td>NA</td></tr><tr><td>3</td><td>Reed et al.<br>(2009)</td><td>Fragility<br>Quality</td><td>DES<br>PoES</td><td>yes</td><td>No</td><td>interdependency matrix between the various subsystems; disturbance or perturbation; wind speed for the hurricane; time in days post-event</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>4</td><td>Carvalho<br>et al. (2008)</td><td>Safety</td><td>PrES</td><td>No</td><td>yes</td><td>possibilities for Micro incidents to escalate and reach negative consequences (loss of control, incidents or accidents); actions under operators' control (use of procedures, training, equipment)</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>5</td><td>Ip and Wang<br>(2011)</td><td>Recovery ability,<br>Performance,<br>Friability</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>the number of reliable passageways between any pair of nodes; reduction in network performance caused by the removal of nodes or edges (based on performance)</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>6</td><td>Morlok and<br>Chang<br>(2004)</td><td>Capacity,<br>Flexibility</td><td>NA</td><td>No</td><td>No</td><td>traffic flow; cost of operation; Traffic Pattern Deviation</td><td>21</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>7</td><td>Sun et al.<br>(2006)</td><td>Degraded-<br>condition,<br>Performance,<br>Capacity,<br>Flexibility</td><td>NA</td><td>No</td><td>No</td><td>number of empty containers per time period; number of loaded containers per time period; number of trains per time period; number of carloads per time period; number of loaded containers per time period for origin-destination (O-D) pairs</td><td>21</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>8</td><td>Cox et al.<br>(2011)</td><td>Vulnerability,<br>Flexibility,<br>Resource<br>availability</td><td>DES<br>PoES</td><td>No</td><td>yes</td><td>the percentage of maximum disruption avoided by resilience behaviours (sub-indictor: Changes in passenger journeys)</td><td>2.20.21</td><td>NA</td><td>NA</td></tr><tr><td>9</td><td>Omer et al.<br>(2009)</td><td>NA</td><td>DES</td><td>No</td><td>No</td><td>the total demand of the node (sub-indictor: total information that needs to be carried through the network to the node); the total information loss</td><td>14</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>10</td><td>Cavalieri<br>et al. (2012)</td><td>Performance,<br>Emergency<br>response planning,<br>Preparedness,<br>Vulnerability</td><td>DES</td><td>yes</td><td>yes</td><td>number of casualties and fatalities; number of displaced people</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>11</td><td>Ouyang and<br>Duenas-<br>Osorio,<br>(2012-b.)</td><td>Performance</td><td>DES<br>PoES</td><td>No</td><td>No</td><td>time; function</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>12</td><td>Li and Lence<br>(2007)</td><td>Reliability,<br>Performance<br>function</td><td>PoES</td><td>No</td><td>No</td><td>the number of failures; the number of successes</td><td>A</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>13</td><td>Miller-Hooks<br>et al. (2012)</td><td>Performance</td><td>PrES<br>DES<br>PoES</td><td>No</td><td>No</td><td>change of demand for O-D pair (sub-indictor: original pre-disaster demand for O-D pair; the post-disaster maximum demand that can be satisfied for O-D pair)</td><td>21</td><td>Quantitative</td><td>Quantitative</td></tr><tr><td>14</td><td>Gilbo (1993)</td><td>Capacity</td><td>NA</td><td>No</td><td>No</td><td>number of arrivals and departures at the airport during a fixed time</td><td>2</td><td>Quantitative</td><td>NA</td></tr><tr><td>15</td><td>Winkler<br>et al. (2010)</td><td>Fragility (of<br>component),<br>Performance</td><td>NA</td><td>No</td><td>No</td><td>the order or number of generators (power plants) and substations, and the size or number of substations, generator edges (transmission lines) for each network; total edge length/</td><td>11</td><td>Quantitative</td><td>Quantitative</td></tr></tbody></table>

(continued on next page)
<table><caption>**Table 7 (continued)**</caption><thead><tr><th rowspan="2">N°</th><th rowspan="2">References</th><th rowspan="2">I<br>Capabilities<br>or properties</th><th rowspan="2">II<br>Temporal<br>stages</th><th rowspan="2">III<br>Interdependence</th><th rowspan="2">IV<br>Implementing<br>action</th><th rowspan="2">V<br>Indicators</th><th rowspan="2">VI<br>CI<br>Sectors</th><th colspan="2">VII<br>Assessment methods</th><th rowspan="2">VIII<br>Indicator</th></tr><tr><th>Resilience</th><th>Indicator</th></tr></thead><tbody><tr><td>16</td><td>Chang<br>(2003)</td><td>Cost-benefit</td><td>DES<br>PoES<br>NEPS</td><td>No</td><td>yes</td><td>number of substation-substation edges, average substation degree, Generators<br>total life cycle costs (sub-indictor: planned costs undertaken by the lifeline agency (maintenance investments and mitigation investment costs), any associated costs imposed on society (e.g., societal disruptions related to maintenance and mitigation activities), expected costs from seismic events (repair costs and revenue loss), the associated expected costs imposed on society (direct economic losses or business interruption<sup>a</sup>)</td><td>A</td><td>Quantitative</td><td>Quantitative</td><td></td></tr></tbody></table>

# 7. Discussion

## 7.1. Necessity of an indicators system

The analysis in this study helps find a significant problem with the existing indicators of CI resilience: existing indicators do not assess resilience according to a uniform and well-defined indicator system. Indicator-based assessment needs an indicators system Shavelson, 1987; Shavelson et al, 1990; UNESCO-LearningPortal). More than just a collection of indicator statistics, indicator systems are usually designed to generate more and more accurate information about conditions Shavelson et al, 1990. Ideally, an indicator system will provide a framework for focus aspects, criteria, and how individual indicator components work together Eurostat, 2014; UNESCO-LearningPortal). However, as can be seen from the results, although these identified indicators are all designed to assess the CI resilience, the indicators systems used by reviewed papers are different due to different indicators, criteria, and assessment methods. In addition, Indicators and criteria are frequently misused. In most of the papers, the resilience assessment is incomplete, as they assess CI's resilience without consideration of various capabilities, and all stages, especially without interdependencies and implementing actions. The problems described above are quite common in indicator-based assessment for CI resilience, which directly leads to results that are not comparable or comprehensive. Comparable assessment results require that the assessment process for different CIs follow uniform criteria and metrics. Comprehensive assessment needs consideration on as many aspects as possible. The establishment of an indicators system aiming at these problems would help improve the indicator-based assessment for CIs resilience.

## 7.2. Limit of aspects

This paper is constructed according to the concepts and aspects presented in part 3. These approaches are presented from a scientific position, founded on a hypothesis that all the aspects presented are compatible with each other. However, these aspects may have a conceptual inconsistency.

To understand and model complex systems, the concept of resilience is often associated with the need for much less centralised control-based, deterministic, and linear approaches. The model grounded on a timeline of a linear transition between stages or levels of system performance or function is one of the examples. However, several mentioned capabilities, such as the capacities in the face of uncertainty and complexity could be contradictory to linear and deterministic models. Being in a complex environment, the performance or function of the infrastructure

may be constantly changing at all stages, and is difficult to be presented as an ideal single linear. In addition, for many researchers, function or performance is no longer a priority property for studying the resilience of CIs. Many properties or capabilities that are not based on simple linearity are becoming increasingly popular in complex systems, such as reliability, resource (cost)-effectiveness, and safety culture.

Nevertheless, it is undeniable that deterministic and linear approaches occupy a very important place in resilience studies. Moreover, the opinion, that deterministic and linear approaches could be combined with the capability in the face of uncertainty, is also accepted by many studies.

## 7.3. Use of results

Meanwhile, this work establishes an initial mapping of resilience indicators and builds a state of the art. For each indicator, this study defines:

* the temporal stage of resilience as noted by PrES (Pre-Event Stage), DES (During Event Stage), PoES (Post-Even Stage) and NEPS (New Event Preparation Stage);

* the suitable CI sectors of indicators;

* the resilience and assessment approaches of indicators.

This type of identification allows stakeholders and scientists to quickly select applicable indicators for their situations and find desired assessment methods through the papers reviewed in the study. For example, for an assessment aiming at the PrES of resilience scenario, based on the tables and Fig. 6, 18 relevant articles could easily be found based on the tables 5, 6, and 7, and Fig. 6. For a study focusing on transport resilience indicators, 48 relevant papers could also be quickly found, 36 of which propose indicators applicable to all CI sectors. Moreover, based on the results, future studies could establish an indicator system at a relatively higher level by observing important areas and existing shortcomings.

## 7.4. Limitation of literature review

The limitations of this study concern the literature review and the elements that aren't viable. Firstly, the "reference search method" is applied by noting only the articles in the existing scientific bibliographies. This led to a lack of investigation of the articles published after the three original review papers Table 3. Secondly, the information about capabilities or properties in several papers is missing (noted with NA in Table 5, 6 and 7). For some papers because the article does not address
<table><caption>**Table A1**

Appendix 1: Papers Selected Through First Reference Search.</caption><thead><tr><th rowspan="2">N°<br>a</th><th rowspan="2">Source</th><th rowspan="2">Title</th><th colspan="3">step: Keyword for Screening</th><th rowspan="2">Step: Abstract screening</th></tr><tr><th>Type 1</th><th>Type 2</th><th>Type 3</th></tr></thead><tbody><tr><td>1</td><td>Shafieezadeh and Ivey Burden, (2014)</td><td>Scenario-based resilience assessment framework for critical infrastructure systems: case study for seismic resilience of seaports.</td><td>resilience</td><td>Critical infrastructure/seaport</td><td>assessment</td><td>X</td></tr><tr><td></td><td>Vugrin and Camphouse (2011-a)</td><td>Infrastructure resilience assessment through control design.</td><td>resilience</td><td>infrastructure</td><td>assessment</td><td>No indicator</td></tr><tr><td></td><td>MacKenzie and Barker (2013)</td><td>Empirical data and regression analysis for estimation of infrastructure resilience with applications to electric power outages</td><td>resilience</td><td>Infrastructure, electric power</td><td>estimation</td><td>No indicator</td></tr><tr><td>2</td><td>Ouyang and Wang (2015)</td><td>Resilience assessment of interdependent infrastructure systems: with a focus on joint restoration modeling and analysis</td><td>resilience</td><td>infrastructure systems</td><td>assessment</td><td>X</td></tr><tr><td>3</td><td>Ouyang and Duenas-Osorio, (2014)</td><td>Multi-dimensional hurricane resilience assessment of electrical power systems</td><td>resilience</td><td>electrical power system</td><td>assessment</td><td>X</td></tr><tr><td>4</td><td>Vugrin et al., (2010)</td><td>A framework for assessing the resilience of infrastructure and economic systems</td><td>resilience</td><td>infrastructure</td><td>assessing</td><td>X</td></tr><tr><td>5</td><td>Shirali et al. (2012)</td><td>Assessing resilience engineering based on safety culture and managerial factors.</td><td>Resilience, safety</td><td>engineering</td><td>assessing</td><td>X</td></tr><tr><td>6</td><td>Shirali et al. (2013)</td><td>A new method for quantitative assessment of resilience engineering by PCA and NT approach: a case study in a process industry.</td><td>resilience</td><td>engineering</td><td>assessment</td><td>X</td></tr><tr><td>7</td><td>Adams et al. (2012)</td><td>Freight Resilience Measures.</td><td>resilience</td><td>Freight System</td><td>measure, Metrics, quantitative</td><td>X</td></tr><tr><td>8</td><td>Henry and Emmanuel Ramirez-Marquez, (2012)</td><td>Generic metrics and quantitative approaches for system resilience as a function of time</td><td>resilience</td><td>network</td><td>measure</td><td>X</td></tr><tr><td></td><td>Barker et al. (2013)</td><td>Resilience-based network component importance measure.</td><td>resilience</td><td>container terminal</td><td>measure</td><td>Measuring the importance of components</td></tr><tr><td>9</td><td>Pant et al. (2014)</td><td>Stochastic measures of resilience and their application to container terminals.</td><td>resilience</td><td>inland waterway network</td><td>measure</td><td>X</td></tr><tr><td></td><td>Baroud et al., (2014a)</td><td>Importance measures for inland waterway network resilience.</td><td>resilience</td><td>network</td><td>measure</td><td>Measuring the importance of critical waterway links</td></tr><tr><td>10</td><td>Baroud et al., (2014b)</td><td>Stochastic Measures of Network Resilience: Applications to Waterway Commodity Flows.</td><td>resilience</td><td>Network</td><td>measuring</td><td>X</td></tr><tr><td></td><td>Wang et al., (2010)</td><td>Measurement of resilience and its application to enterprise information systems.</td><td>resilience</td><td>system</td><td>Measurement</td><td>No suitable for infrastructures</td></tr><tr><td>11</td><td>Chen and Miller-Hooks (2012)</td><td>Resilience: an indicator of recovery capability in intermodal freight transport.</td><td>Resilience, recovery capacity</td><td>intermodal freight transport</td><td>indicator</td><td>X</td></tr><tr><td></td><td>Janic (2015)</td><td>Modeling the resilience, friability and costs of an air transport network affected by a large-scale disruptive event</td><td>resilience, friability and costs</td><td>transport network</td><td>Modelling</td><td>No indicator</td></tr><tr><td>12</td><td>Ayyub, (2014)</td><td>Systems resilience for multihazard environments: definition, metrics, and valuation for decision making</td><td>resilience</td><td>system</td><td>metric</td><td>X</td></tr><tr><td>13</td><td>Hashimoto et al., (1982)</td><td>Reliability, resiliency, and vulnerability criteria for water resource system performance evaluation</td><td>Reliability, resiliency, vulnerability, performance</td><td>water resource system</td><td>Criteria, Evaluation</td><td>X</td></tr><tr><td>14</td><td>Franchin and Cavalieri, (2015)</td><td>Probabilistic assessment of civil infrastructure resilience to earthquakes</td><td>resilience</td><td>infrastructure</td><td>Assessment</td><td>X</td></tr><tr><td></td><td>Attoh-Okine et al., (2009)</td><td>Formulation of resilience index of urban infrastructures using belief functions</td><td>resilience</td><td>Urban infrastructure</td><td>index</td><td>No indicator</td></tr><tr><td>15</td><td>Faturechi et al. (2014)</td><td>Evaluating and optimizing resilience of airport pavement networks</td><td>resilience</td><td>resilience</td><td>Evaluating</td><td>X</td></tr><tr><td>16</td><td>Faturechi and Miller-Hooks (2014)</td><td>Travel time resilience of roadway networks under disaster</td><td>resilience</td><td>Road network</td><td>Travel time</td><td>X (travail time)</td></tr><tr><td></td><td>Azadeh et al. (2014)</td><td>Performance evaluation of integrated resilience engineering factors by data envelopment analysis: the case of a petrochemical plant</td><td>performance</td><td>Engineering, petrochemical plant</td><td>Evaluation</td><td>Assessing the reliability of indicators, not the resilience</td></tr><tr><td></td><td>Cardoso et al. (2014)</td><td>Resilience assessment of supply chains under different types of disruption</td><td>resilience</td><td>supply chains</td><td>assessment</td><td>supply chains are not infrastructures</td></tr><tr><td></td><td>Khaled et al. (2015)</td><td>Train design and routing optimization for evaluating criticality of freight railroad infrastructures</td><td>Criticality</td><td>freight railroad infrastructures</td><td>Evaluation</td><td>Identifying critical rail infrastructures</td></tr><tr><td></td><td>(Spiegler et al., 2012)</td><td>A control engineering approach to the assessment of supply chain resilience</td><td>resilience</td><td>supply chain, engineering</td><td>assessment</td><td>supply chains are not infrastructures</td></tr></tbody></table>

(continued on next page)
<table><caption>Table A1 (continued)</caption><thead><tr><th rowspan="2">N°<br>a</th><th rowspan="2">Source</th><th rowspan="2">Title</th><th colspan="3">step: Keyword for Screening</th><th rowspan="2">Step: Abstract screening</th></tr><tr><th>Type 1</th><th>Type 2</th><th>Type 3</th></tr></thead><tbody><tr><td rowspan="3">17</td><td>Adjetey-Bahun et al. (2014)</td><td>Planchet, A simulation-based approach to quantifying resilience indicators in a mass transportation system</td><td>resilience</td><td>transportation system</td><td>Indicator, quantifying</td><td>X</td></tr><tr><td>Azadeh et al. (2014)</td><td>Assessment of resilience engineering factors in high-risk environments by fuzzy cognitive maps: a petrochemical plant</td><td>resilience</td><td>Engineering, petrochemical plant</td><td>assessment</td><td>supply chains are not infrastructures</td></tr><tr><td>Muller (2012)</td><td>Fuzzy architecture assessment for critical infrastructure resilience</td><td>resilience</td><td>Critical infrastructure</td><td>assessment</td><td>presents an approach for the selection of alternative architectures in a connected infrastructure system</td></tr><tr><td rowspan="2">18</td><td>Enjalbert et al. (2011)</td><td>Assessment of transportation system resilience</td><td>Resilience</td><td>Transportation system</td><td>assessment</td><td>X</td></tr><tr><td>Francis and Bekera (2014)</td><td>A metric and frameworks for resilience analysis of engineered and infrastructure systems</td><td>resilience</td><td>engineered and infrastructure system</td><td>metric</td><td>No indicator</td></tr><tr><td rowspan="3">19</td><td>Curt et al., (2010)</td><td>A knowledge formalization and aggregation-based method for the assessment of dam performance</td><td>performance</td><td>dam</td><td>assessment</td><td>X</td></tr><tr><td>Petit et al., (2013)</td><td>Resilience Measurement Index: an indicator of critical infrastructure resilience.</td><td>resilience</td><td>Critical infrastructure</td><td>Measurement, index, indicator</td><td>No indicator</td></tr><tr><td>Rosati and Touzinsky, (2015)</td><td>Quantifying coastal system resilience for the US Army Corps of Engineers</td><td>resilience</td><td>coastal system</td><td>Quantifying</td><td>Community resilience</td></tr><tr><td>20</td><td>Lhomme et al. (2013)</td><td>Urban technical networks resilience assessment</td><td>resilience</td><td>Technical network</td><td>assessment</td><td>X</td></tr></tbody></table>

the information sought, and for others because the message is not clear. In the spirit of furthering this work, the future idea could be to formalise a criterion and analyse all studies against this, eventually filling the “NA” in Fig. 6. Moreover, the future study could also suggest possible assessment methods for the articles that do not address the information intended. Table 4.

## 8. Conclusion

To investigate the existing resilience indicators for CIs, this paper undertook the following steps:

- define the elements to be identified for this study by presenting current studies for indicator-based assessment of CI resilience.

- screen the relevant scientific papers with two search methods, including the method for indicators identification;

- analyse the target elements based on results.

The results of the study contribute to a comprehensive understanding and application of resilience indicators for CIs. Meanwhile, it provides a clear orientation for further research (i.e., establishing an indicator system for CI resilience assessment). For this goal, several questions are highlighted based on the results of this review work:

* since the results show that indicators can be both general and specific, how develop general indicators systems that could be suitable for all CIs, at the same time specific and practical for applying to different CIs and contexts? Should the indicator system be classified into several levels for this issue?

* how to develop the general indicator systems that could be suitable for all CIs, which are also specific and practical for applying to different CIs and contexts? Should the indicator system be classified in several levels for this issue?

* how to construct the indicator systems that take into account implementing actions and interdependencies between CIs, internal components and also between CIs and other urban components?

* how does the indicator system address the shortcomings of existing indicators, like the lack of assessment relating to the Next Event Preparation Stage;

* how to establish efficient criteria for resilience assessment;

* how to formalise the methods for “resilience assessment” and “indicator assessment”;

All of these questions are worth considering in the future. Funding.

The authors performed this work with funding from the UrbRiskLab (URL) project (https://urbarisklab.org/fr/ and the RESIIST project (ANR-18-CE39-0018, https://research.gi.mines-albi.fr/display/resiist/RESIIST + Home (in French)) that is jointly funded by the French National Research Agency (ANR) and the General Secretary of Defence and National Security (SGDSN). The authors acknowledge these organisations for their support that helped improve the paper.

## CRediT authorship contribution statement

**Zhuyu Yang:** Writing – original draft, Visualization, Methodology, Investigation, Formal analysis, Conceptualization. **Bruno Barroca:** Writing – review & editing, Supervision, Methodology, Conceptualization. **Alexandre Weppe:** Writing – original draft, Visualization, Investigation. **Aurélia Bony-Dandrieux:** Writing – review & editing, Supervision. **Katia Laffréchine:** Writing – review & editing, Supervision. **Nicolas Daclin:** Writing – review & editing, Supervision. **Valérie November:** Validation. **Khaled Omrane:** Validation. **Daouda Kamisoko:** Funding acquisition. **Frederick Benaben:** Validation, Methodology. **Hélène Dolidon:** Validation. **Jérôme Tixier:** Validation. **Vincent
<table><caption>**Table A2**
Appendix 2: Papers Selected Through Second Reference Search.</caption><thead><tr><th rowspan="2">N°<br>b</th><th rowspan="2">Source</th><th rowspan="2">Title</th><th colspan="3">Step 3: Keyword for Screening</th><th rowspan="2">Step 4: Abstract<br>screening</th></tr><tr><th>Type 1</th><th>Type 2</th><th>Type 3</th></tr></thead><tbody><tr><td>1</td><td>Jovanović et al.<br>(2018)</td><td>An indicator-based approach to assessing resilience of smart critical infrastructures</td><td>resilience</td><td>critical infrastructure</td><td>assessing</td><td>No source</td></tr><tr><td>2</td><td>Serre (2018)</td><td>DS3 Model Testing: Assessing Critical Infrastructure Network Flood Resilience at the Neighbourhood Scale</td><td>resilience</td><td>Critical Infrastructure</td><td>assessing</td><td>X</td></tr><tr><td>3</td><td>Fisher et al.<br>(2010)</td><td>Constructing a resilience index for the enhanced critical infrastructure protection program</td><td>resilience</td><td>critical infrastructure</td><td>index</td><td>X</td></tr><tr><td>3</td><td>Reed et al. (2009)</td><td>Methodology for assessing the resilience of networked infrastructure</td><td>resilience</td><td>infrastructure</td><td>assessing</td><td>X</td></tr><tr><td>3</td><td>Khazai et al.<br>(2013)</td><td>An Integrated Indicator Framework for Spatial Assessment of Industrial and Social Vulnerability to Indirect Disaster Losses</td><td>Vulnerability</td><td>Industrial</td><td>Assessment</td><td>Social community resilience</td></tr><tr><td>3</td><td>Molyneaux et al.<br>(2016)</td><td>Measuring Resilience in Energy Systems: Insights from a Range of Disciplines</td><td>Resilience</td><td>Energy System</td><td>measuring</td><td>Assess security of oil supplies which is not infrastructure</td></tr><tr><td>3</td><td>Vugrin et al.,<br>(2011)</td><td>A resilience assessment framework for infrastructure and economic systems: Quantitative and qualitative resilience analysis of petrochemical supply chains to a hurricane</td><td>Resilience</td><td>Infrastructure, petrochemical supply chains</td><td>assessment</td><td>Not for CIs</td></tr><tr><td>4</td><td>Costella et al.<br>(2009)</td><td>A method for assessing health and safety management systems from the resilience engineering perspective</td><td>Safety, resilience</td><td>System</td><td>Assessment</td><td>health and safety management systems</td></tr><tr><td>4</td><td>Carvalho et al.<br>(2008)</td><td>Micro incident analysis framework to assess safety and resilience in the operation of safe critical systems: a case study in a nuclear power plant.</td><td>Safety, resilience</td><td>Systems, nuclear power plant</td><td>assess</td><td>X</td></tr><tr><td>4</td><td>Saurin et al.,<br>(2011)</td><td>Evaluation and improvement of a method for assessing HSMS from the resilience engineering perspective: a case study of an electricity distributor</td><td>Resilience</td><td>electricity distributor, engineering</td><td>Evaluation, assessing</td><td>No infrastructure</td></tr><tr><td>5</td><td>Ip and Wang<br>(2011)</td><td>Resilience and friability of transportation networks: evaluation, analysis and optimization</td><td>Resilience, friability</td><td>transportation networks</td><td>Evaluation</td><td>x</td></tr><tr><td>5</td><td>Miller et al.<br>(2004)</td><td>Evaluation of network reliability calculation methods</td><td>Reliability</td><td>network</td><td>Evaluation</td><td>It discusses network, not infrastructure</td></tr><tr><td>5</td><td>Dutuit and Rauzy,<br>(2005)</td><td>Approximate estimation of system reliability via fault trees.</td><td>Reliability</td><td>system</td><td>Estimation</td><td>It discusses network, not infrastructure</td></tr><tr><td>5</td><td>Zhang et al.<br>(2011)</td><td>A new holistic method for reliability performance assessment and critical components detection in complex networks.</td><td>reliability performance</td><td>network</td><td>Assessment</td><td>No indicator</td></tr><tr><td>5</td><td>Kalyoncu and<br>Sankur (1992)</td><td>Estimation of survivability of communication networks.</td><td>survivability</td><td>communication network</td><td>Estimation</td><td>No resource</td></tr><tr><td>5</td><td>Cho (2002)</td><td>Three papers on measuring the reliability and flexibility of transportation system capacity</td><td>Reliability, flexibility, capacity</td><td>transportation system</td><td>Measuring</td><td>No resource</td></tr><tr><td>6</td><td>Morlok and<br>Chang (2004)</td><td>Measuring capacity flexibility of a transportation system</td><td>capacity flexibility</td><td>transportation system</td><td>Measuring</td><td>Socio-economic system: transportation network service</td></tr><tr><td>6</td><td>Najjar and<br>Gaudiot, (1990)</td><td>Network resilience: A measure of network fault tolerance.</td><td>Resilience</td><td>Network</td><td>measure</td><td>No suitable for infrastructures</td></tr><tr><td>6</td><td>Srinivasan (2002)</td><td>Transportation network vulnerability assessment: A quantitative framework.</td><td>vulnerability</td><td>Transportation network</td><td>Assessment</td><td>No indicator</td></tr><tr><td>7</td><td>Sun et al. (2006)</td><td>Estimating freight transportation system capacity, flexibility, and degraded-condition performance</td><td>Capacity, performance</td><td>freight transportation system</td><td>Estimating</td><td>x</td></tr><tr><td>8</td><td>Cox et al. (2011)</td><td>Transportation security and the role of resilience: A foundation for operational metrics</td><td>Resilience</td><td>Transportation</td><td>metrics</td><td>X</td></tr><tr><td>8</td><td>Garbin, and<br>Shortle (2007)</td><td>Measuring Resilience in Network-Based Infrastructures</td><td>resilience</td><td>infrastructure</td><td>Measuring</td><td>No resource</td></tr><tr><td>9</td><td>Omer et al.,<br>(2009)</td><td>Measuring the Resilience of the Global Internet Infrastructure System</td><td>resilience</td><td>Infrastructure</td><td>Measuring</td><td>x</td></tr><tr><td>10</td><td>Cavalieri et al.<br>(2012)</td><td>Quantitative assessment of social losses based on physical damage and interaction with infrastructural systems</td><td>damage, loss</td><td>Infrastructural system</td><td>Assessment</td><td>x</td></tr><tr><td>11</td><td>Ouyang and<br>Duenas-Osorio,<br>(2012-b.)</td><td>Time-dependent resilience assessment and improvement of urban infrastructure systems</td><td>resilience</td><td>infrastructure</td><td>Assessment</td><td>x</td></tr><tr><td>12</td><td>Li and Lence,<br>(2007)</td><td>Estimating Resilience of Water Resources Systems</td><td>Resilience</td><td>Water Resources Systems</td><td>Estimating</td><td>X</td></tr><tr><td>13</td><td>Miller-Hooks<br>et al. (2012)</td><td>Measuring and maximizing resilience of freight transportation networks</td><td>resilience</td><td>networks</td><td>Measuring</td><td>X</td></tr><tr><td>13</td><td>Xu (2009)</td><td>Evaluating sustainability and resilience of complex systems using network theory.</td><td>sustainability and resilience</td><td>complex systems</td><td>Evaluating</td><td>No resource</td></tr><tr><td>14</td><td>Gilbo (1993)</td><td>Airport capacity: representation, estimation, optimization.</td><td>capacity</td><td>Airport</td><td>estimation</td><td>X</td></tr><tr><td>15</td><td>Winkler et al.,<br>2010</td><td>Performance assessment of topologically diverse power systems subjected to hurricane events.</td><td>Performance</td><td>diverse power systems</td><td>assessment</td><td>X</td></tr><tr><td>15</td><td>Chen et al. (2005)</td><td>Cascading dynamics and mitigation assessment in power system disturbance via a hidden failure model</td><td>Cascading dynamics and mitigation</td><td>power system</td><td>assessment</td><td>Not related to resilience</td></tr></tbody></table>

(continued on next page)
<table><caption>Table A2 (continued)</caption><thead><tr><th rowspan="2">N°<br>b</th><th rowspan="2">Source</th><th rowspan="2">Title</th><th colspan="3">Step 3: Keyword for Screening</th><th rowspan="2">Step 4: Abstract<br>screening</th></tr><tr><th>Type 1</th><th>Type 2</th><th>Type 3</th></tr></thead><tbody><tr><td>16</td><td>Chang (2003)<br>Fisher et al.<br>(2009)</td><td>Evaluating disaster mitigations: methodology for urban infrastructure systems.<br>Constructing Vulnerability and Protective Measures Indices for the Enhanced Critical Infrastructure Protection Program</td><td>disaster mitigation<br>Vulnerability</td><td>infrastructure<br>Critical Infrastructure</td><td>Evaluation<br>Measure, indices,</td><td>X<br>No indicator</td></tr></tbody></table>

## Chapurlat: Validation.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data availability

No data was used for the research described in the article.

## References

* National Academies of Sciences, Engineering, and Medicine. 1992. Assessment of the U. S. Outer Continental Shelf Environmental Studies Program: III. Social and Economic Studies. Washington, DC: The National Academies Press. https://doi.org/10.17226/2062.

* Adams, T.M., Bekkem, K.R., Toledo-Durán, E.J., 2012. Freight resilience measures. J. Transp. Eng. 138, 1403–1409. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000415.

* Adjeyte-Bahun, K., Birregah, B., Chátelet, E., Planchet, J.L., Laurens-Fonseca, E., 2014. A simulation-based approach to quantifying resilience indicators in a mass transportation system. ISCRAM 2014 Conf. Proc. - 11th Int. Conf. Inf. Syst. Crisis. Response Manag. 75–79.

* Aldeberky, A.A., 2007. The influence of high-rise buildings on the environment. Atmospheric Environment 180–191.

* Alderson, D.L., Brown, G.G., Carlyle, W.M., 2015. Operational models of infrastructure resilience. Risk Analysis 35 (4), 562–586.

* Argyroudis, S.A., Mitoulis, S.A., Winter, M.G., Kaynia, A.M., 2019. Fragility of transport assets exposed to multiple hazards: State-of-the-art review toward infrastructural resilience. Reliability Engineering & System Safety 191, 106567.

* Atkinson, R., Crawford, L., Ward, S., 2006. Fundamental uncertainties in projects and the scope of project management. International journal of project management 24 (8), 687–698.

* Attoh-Okine, N.O., Cooper, A.T., Mensah, S.A., 2009. Formulation of resilience index of urban infrastructure using belief functions. IEEE Systems Journal 3 (2), 147–153.

* Ayyub, B.M., 2014. Systems resilience for multihazard environments: Definition, metrics, and valuation for decision making. Risk Analysis 34 (2), 340–355.

* Azadeh, A., Salehi, V., Arvan, M., Dolatkhah, M., 2014. Assessment of resilience engineering factors in high-risk environments by fuzzy cognitive maps: A petrochemical plant 68, 99–107.

* Bambara, G., Peyras, L., Felix, H., Serre, D., 2015. Developing a functional model for cities impacted by a natural hazard: application to a city affected by flooding. Natural hazards and earth system sciences 15 (3), 603–615.

* Barker, K., Ramirez-Marquez, J.E., Rocco, C.M., 2013. Resilience-based network component importance measures. Reliability Engineering & System Safety 117, 89–97.

* Baroud, H., Barker, K., Ramirez-Marquez, J.E., S. c.m.r., 2014a. Importance measures for inland waterway network resilience 62, 55–67.

* Baroud, H., Ramirez-Marquez, J.E., Barker, K., Rocco, C.M., 2014b. Stochastic Measures of Network Resilience: Applications to Waterway Commodity Flows. Risk Anal. 34, 1317–1335. https://doi.org/10.1111/risa.12175.

* Barroca, B., Serre, D., Youssef, D., 2012. Le concept de résilience à l'épreuve du génie urbain Le concept de résilience à l'épreuve du génie urbain. https://doi.org/10.4000/vertigo.12469.

* Barroca, B., Serre, D., 2013. Behind the barriers: a resilience conceptual model. SAPI EN. S. Surveys and Perspectives Integrating Environment and Society 6.1.

* Bescos, P.L., Dobler, P., Mendoza, C., Naulleau, G., Giraud, F., Anger, V.L., 1997. Contrôle de gestion et management. Montchrestien.

* Bialas, A., 2016. Critical Infrastructure Protection—How to Assess the Protection Efficiency. In: International Conference on Dependability and Complex Systems. Springer, Cham, pp. 25–37.

* Bourgeois, D., 2014. Information systems for business and beyond. The Saylor Foundation.

* Brown, G., Carlyle, M., Salmerón, J., Wood, K., 2006. Defending critical infrastructure. Interfaces (Providence). 36, 530–544. https://doi.org/10.1287/inte.1060.0252.

* Bruneau, M., Chang, S.E., Eguchi, R.T., Lee, G.C., O'Rourke, T.D., Reinhorn, A.M., Von Winterfeldt, D., 2003. A framework to quantitatively assess and enhance the seismic resilience of communities. Earthquake spectra 19 (4), 733–752.

* Buesseler, K., Dai, M., Aoyama, M., Benitez-Nelson, C., Charmasson, S., Higley, K., Smith, J.N., 2017. Fukushima Daichi-derived radionuclides in the ocean: transport, fate, and impacts. Annual review of marine science 9, 173–203.

* Cantelmi, R., Di Gravio, G., Patriarca, R., 2021. Reviewing qualitative research approaches in the context of critical infrastructure resilience, Environment Systems and Decisions. Springer US. https://doi.org/10.1007/s10669-020-09795-8.

* Cardoso, S.R., Barbosa-póvoa, A.P.F.D., Relvas, S., 2014. Resilience assessment of supply chains under different types of disruption. https://doi.org/10.1016/B978-0-444-63433-7.50111-5.

* Carvalho, P.V.R., dos Santos, I.L., Gomes, J.O., Borges, M.R.S., 2008. Micro incident analysis framework to assess safety and resilience in the operation of safe critical systems: A case study in a nuclear power plant. J. Loss Prev. Process Ind. 21, 277–286. https://doi.org/10.1016/j.jlpl.2007.04.005.

* Cavalieri, F., Franchin, P., Gehl, P., Khazai, B., 2012. Quantitative assessment of social losses based on physical damage and interaction with infrastructural systems. Earthquake Engineering & Structural Dynamics 41 (11), 1569–1589.

* Cerema, 2020. La boussole de la résilience: repères pour la résilience territorial.

* Chang, S.E., 2003. Evaluating disaster mitigations: Methodology for urban infrastructure systems. Natural Hazards Review 4 (4), 186–196.

* Chang, S. E., 2016. Socioeconomic impacts of infrastructure disruptions. In Oxford research encyclopedia of natural hazard science.

* Chen, L., Miller-Hooks, E., 2012. Resilience: An indicator of recovery capability in intermodal freight transport. Transp. Sci. 46, 109–123. https://doi.org/10.1287/trsc.1110.0376.

* Chen, J., Thorp, J.S., Dobson, I., 2005. Cascading dynamics and mitigation assessment in power system disturbances via a hidden failure model 27, 318–326. https://doi.org/10.1016/j.ijepes.2004.12.003.

* Cho, D.J., 2002. Three papers on measuring the reliability and flexibility of transportation system capacity. University of. Pennsylvania.

* Cimellaro, G.P., Tinebra, A., Renschler, C., Fragiadakis, M., 2016. New Resilience Index for Urban Water Distribution Networks. J. Struct. Eng. 142, 1–13. https://doi.org/10.1061/(asce)st.1943-541x.0001433.

* CIIPedia, https://websites.fraunhofer.de/CIPedia/index.php/Critical_Infrastructure, 2020. (accessed 11 May 2022).

* European Commission, 2005. Green Paper on a European programme for critical infrastructure protection, https://op.europa.eu/en/publication-detail/-/publication/4e3f9be0-ce1c-4f5c-9fdc-07bdd441fb88/language-en (accessed 27 September 2022).

* CORDIS-IMPROVER Project. Report on Current Approaches for Defining, Evaluating and Implementing Resilience Concepts in Critical Infrastructure, https://cordis.europa.eu/project/id/653390/results, 2016 (accessed 13 March 2022).

* CORDIS-Smart Resilience Indicators for Smart Critical Infrastructures, Resilience indicators for SCIs based on big and open data. https://cordis.europa.eu/project/id/700621/results, 2018 (accessed 11 May 2022).

* Costella, M.F., Saurin, T.A., de Macedo Guimaraes, L.B., 2009. A method for assessing health and safety management systems from the resilience engineering perspective. Safety Science 47 (8), 1056–1067.

* Cox, A., Prager, F., Rose, A., 2011. Transportation security and the role of resilience: A foundation for operational metrics. Transp. Policy 18, 307–317. https://doi.org/10.1016/j.tranpol.2010.09.004.

* Curt, C., Peyras, L., Boissier, D., 2010. A knowledge formalization and aggregation-based method for the assessment of dam performance. Comput. Civ. Infrastruct. Eng. 25, 171–184. https://doi.org/10.1111/j.1467-8667.2009.00604.x.

* Curt, C., Tacnet, J.M., 2018. Resilience of Critical Infrastructures: Review and Analysis of Current Approaches. Risk Anal. 38, 2441–2458. https://doi.org/10.1111/risa.13166.

* Cutter, S.L., 2016. The landscape of disaster resilience indicators in the USA. Natural hazards 80 (2), 741–758.

* Cutter, S.L., Burton, C.G., Emrich, C.T., 2010. Disaster resilience indicators for benchmarking baseline conditions. Journal of homeland security and emergency management 7 (1).

* De Vivo, C., Ellena, M., Capozzi, V., Budillon, G., Mercogliano, P., 2022. Risk assessment framework for Mediterranean airports: a focus on extreme temperatures and precipitations and sea level rise. Nat. Hazards 111, 547–566. https://doi.org/10.1007/s11069-021-05066-0.

* Duffaut, P., 2013. The traps behind the failure of Malpasset arch dam, France, in 1959. Journal of Rock Mechanics and Geotechnical Engineering 5 (5), 335–341.

* Duttuit, Y., Rauzy, A., 2005. Approximate estimation of system reliability via fault trees. Reliability engineering & system safety 87 (2), 163–172.
Dyer, O., 2020. Beirut's battered hospitals struggle in aftermath of explosion. BMJ 370, m3171. https://doi.org/10.1136/bmj.m3171.

Enjalbert, S., Vanderhaegen, F., Pichon, M., 2011. Human Modelling in Assisted Transportation. Hum. Model. Assist. Transp. https://doi.org/10.1007/978-88-470-1821-1.

Eriksson, J., Juhl, K. A., 2012. Guide to risk and vulnerability analyses. Swedish Civil Contingencies Agency (MSB).

EUR-Lex, https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=legissum%3A313259, 2010. (accessed 11 May 2022).

Eurostat, Getting messages across using indicators - A handbook based on experiences from assessing Sustainable Development Indicators, https://ec.europa.eu/eurostat/web/products-manuals-and-guidelines/-/ks-gq-12-001, 2014 (accessed 11 May 2022).

Futurechi, R., Levenberg, E., Miller-Hooks, E., 2014. Evaluating and optimizing resilience of airport pavement networks. Comput. Oper. Res. 43, 335-348. https://doi.org/10.1016/j.cor.2013.10.009.

Futurechi, R., Miller-Hooks, E., 2014. Travel time resilience of roadway networks under disaster. Transp. Res. Part B Methodol. 70, 47-64. https://doi.org/10.1016/j.trb.2014.08.007.

Fisher, R. E., Buehring, W. A., Whitfield, R. G., Bassett, G. W., Dickinson, D. C., Haffenden, R. A., ... Lawlor, M. A., 2009. *Constructing vulnerability and protective measures indices for the enhanced critical infrastructure protection program* (No. ANL/ DIS-09-4). Argonne National Lab.(ANL), Argonne, IL (United States).

Fisher, R. E., Bassett, G. W., Buehring, W. A., Collins, M. J., Dickinson, D. C., Eaton, L. K., ... Peerenboom, J. P., 2010. *Constructing a resilience index for the enhanced critical infrastructure protection program* (No. ANL/DIS-10-9). Argonne National Lab.(ANL), Argonne, IL (United States). Decision and Information Sciences.

Franchin, P., Cavalieri, F., 2015. Probabilistic assessment of civil infrastructure resilience to earthquakes. Comput. Civ. Infrastruct. Eng. 30, 583-600. https://doi.org/10.1111/mice.12092.

Francis, R., Bekera, B., 2014. A metric and frameworks for resilience analysis of engineered and infrastructure systems. Reliab. Eng. Syst. Saf. 121, 90-103. https:// doi.org/10.1016/j.ress.2013.07.004.

Garbin, D. A., Shortle, J. F., 2007. Measuring resilience in network-based infrastructures. *Critical Thinking: Moving from infrastructure protection to infrastructure resilience*, 73- 85.

Gasser, P., Lustenberger, P., Cinelli, M., Kim, W., Spada, M., Burgherr, P., Hirschberg, S., Stojadinovic, B., Sun, T.Y., 2021. A review on resilience assessment of energy systems. Sustain. Resilient Infrastruct. 6, 273-299. https://doi.org/10.1080/ 23789689.2019.1610600.

Ghosn, M., Duenas-Osorio, L., Frangopol, D.M., McAllister, T.P., Bocchini, P., Manuel, L., Ellingwood, B.R., Arangio, S., Bontempi, F., Shah, M., Akiyama, M., Biondini, F., Hernandez, S., Tsiatas, G., 2016. Performance Indicators for Structural Systems and Infrastructure Networks. J. Struct. Eng. 142, 1-18. https://doi.org/10.1061/(asc e)st.1943-541x.0001542.

Gilbo, E.P., 1993. Airport capacity: Representation, estimation, optimization. IEEE Transactions on control systems technology 1 (3), 144-154.

Grimaz, S., Ruzzene, E., Zorzini, F., 2021. Situational assessment of hospital facilities for modernization purposes and resilience improvement. International Journal of Disaster Risk Reduction 66, 102594.

Gyau-Boakye, P., 2001. Environmental impacts of the Akosombo dam and effects of climate change on the lake levels. Environment, Development and Sustainability 3 (1), 17-29.

Hashimoto, T., Stedinger, J.R., Loucks, D.P., 1982. Reliability, resiliency, and vulnerability criteria for water resource system performance evaluation. Water Resour. Res. 18, 14-20. https://doi.org/10.1029/WR018i001p00014.

Heinzlel, C., Barroca, B., Leone, M., Serre, D., 2022. Urban resilience operationalization issues in climate risk management: A review. Int. J. Disaster Risk Reduct. 75, 102974 https://doi.org/10.1016/j.ijdrr.2022.102974.

Henry, D., Emmanuel Ramirez-Marquez, J., 2012. Generic metrics and quantitative approaches for system resilience as a function of time. Reliab. Eng. Syst. Saf. 99, 114-122. https://doi.org/10.1016/j.ress.2011.09.002.

Hills, A., 2005. Insidious environments: creeping dependencies and urban vulnerabilities. Journal of Contingencies and Crisis Management 13 (1), 12-20.

Holling, C.S., 1973. Resilience and stability of ecological systems. Annual review of ecology and systemsatics 4 (1), 1-23.

Hollnagel, E., 2014. Resilience engineering and the built environment. Building Research & Information 42 (2), 221-228.

Hollnagel, E., Woods, D.D., 2017. Epilogue: Resilience engineering precepts. In: Resilience Engineering. CRC Press, pp. 347-358.

Hollnagel, E. (2015). RAG-resilience analysis grid. Introduction to the Resilience Analysis Grid (RAG).

Hosseini, S., Barker, K., Ramirez-Marquez, J.E., 2016. A review of definitions and measures of system resilience. Reliab. Eng. Syst. Saf. 145, 47-61. https://doi.org/10.1016/j.ress.2015.08.006.

Hromada, M., Lukas, L., 2012. Conceptual design of the resilience evaluation system of critical infrastructure elements and networks in selected areas in Czech republic. 2012 IEEE Int. Conf. Technol. Homel. Secur. HST 2012, 353-358. https://doi.org/10.1109/THS.2012.6459874.

Cybersecurity & Infrastructure security Agency (2020). https://www.cisa.gov/critical- infrastructure-sectors, 2022 09 23 (accessed 27 September 2022).

Ip, W.H., Wang, D., 2011. Resilience and friability of transportation networks: Evaluation, analysis and optimization. IEEE Syst. J. 5, 189-198. https://doi.org/10.1109/JSYST.2010.2096670.

ISO (2021), ISO 22300:2021- Security and resilience-Vocabulary, https://www.iso.org/fr/standard/77008.html, Accessed September 14, 2022.

Janic, M., 2015. Reprint of "" Modelling the resilience, friability and costs of an air transport network affected by a large-scale disruptive event "". q 71, 77-92. https:// doi.org/10.1016/j.tra.2015.07.012.

Jeong, S., An, Y.Y., 2016. Climate change risk assessment method for electrical facility. 2016 Int. Conf. Fin. Commun. Technol. Converg. ICTC 2016, 184-188. https://doi. org/10.1109/ICTC.2016.7763464.

Johnsen, S.O., Veen, M., 2013. Risk assessment and resilience of critical communication infrastructure in railways. Cogn. Technol. Work 15, 95-107. https://doi.org/10.1007/s10111-011-0187-2.

Jovanović, A., Oien, K., Choudhary, A., 2018. An indicator-based approach to assessing resilience of smart critical infrastructures. In: Urban Disaster Resilience and Security. Springer, Cham, pp. 285-311.

Kalyoncu, H., Sankur, B., 1992. Estimation of survivability of communication networks. Electronics Letters 28 (19), 1790-1791.

Kamissoko, D., Nastov, B., Chapurlat, V., Dolidon, H., Bony-Dandrieux, A., Barroca, B., Marechal, M., Tixier, J., Allon, M., Benaben, F., Daclin, N., Muller, A., Salatge, N., November, V., 2020. Resilience of Interconnected Infrastructures and Systems: The RESIIST Project. Commun. Comput. Inf. Sci. 1193 CCIS, 257-270. https://doi.org/10.1007/978-3-030-42517-3_20.

Kammouh, O., Cardoni, A., Marasco, S., Cimellaro, G.P., Mahin, S., 2018. Resilience assessment of city-scale transportation networks using Monte Carlo simulation. Maintenance, Safety, Risk, Manag. Life-Cycle Perform. Bridg. - Proc. 9th Int. Conf. Bridg. Maintenance. Saf. Manag. IABMAS 2018, 1902-1910. https://doi.org/10.1201/9781315189390-259.

Khaled, A.A., Jin, M., Clarke, D.B., Hoque, M.A., 2015. Train design and routing optimization for evaluating criticality of freight railroad infrastructures. Transp. Res. Part B 71, 71-84. https://doi.org/10.1016/j.trb.2014.10.002.

Khzai, B., Merz, M., Schulz, C., Borst, D., 2013. An integrated indicator framework for spatial assessment of industrial and social vulnerability to indirect disaster losses 145-167. https://doi.org/10.1007/s11069-013-0551-z.

Kizlik, B., 2012. Measurement, assessment, and evaluation in education.

Kolowrocki, K., 2020. Examination of the safety of a port oil terminal. Sci. Journals Marit. Univ. Szczecin-Zeszyty Nauk. Akad. Morskiej W Szczecinie 61, 143-151. https://doi.org/10.17402/410.

Kolowrocki, K., Soszyńska-budny, J., 2019. Safety and resilience indicators of critical infrastructure impacted by operation application to port oil terminal examination. TransNav 13, 761-769. https://doi.org/10.12716/1001.13.04.08.

Kolowrocki, K., Soszyńska-Budny, J., Torbicki, M., 2018. Critical infrastructure impacted by climate change safety and resilience indicators. In: *In 2018 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM)*. IEEE, pp. 986-990.

Kolowrocki, K., Soszyńska-Budny, J., Torbicki, M., 2019-a. Critical Infrastructure Impacted by Climate Change Safety and Resilience Indicators. IEEE Int. Conf. Ind. Eng. Eng. Manag. 2019-December, 986-990. https://doi.org/10.1109/IEEM.2018. 8607565.

Kolowrocki, K., Soszyńska-Budny, J., Torbicki, M., 2019-b. Critical Infrastructure Impacted by Climate Change Safety and Resilience Indicators. IEEE Int. Conf. Ind. Eng. Eng. Manag. 2019-December, 986-990. https://doi.org/10.1109/ IEEM.2018.8607565.

LaLone, N., Tapia, A., Zobel, C., Caraega, C., Neppalli, V.K., Halse, S., 2017. Embracing human noise as resilience indicator: twitter as power grid correlate. Sustain. Resilient Infrastruct. 2, 169-178. https://doi.org/10.1080/ 23789689.2017.1328920.

Leveson, N., Dulac, N., Zipkin, D., Cutcher-Gershenfeld, J., Carroll, J., Barrett, B., 2017. Engineering resilience into safety-critical systems. In: Resilience engineering. CRC Press, pp. 95-123.

Lhomme, S., Serre, D., Diab, Y., Laganier, R., 2013. Urban technical networks resilience assessment. Resilience and urban risk management 109-117.

Li, Y., Lence, B.J., 2007. Estimating resilience for water resources systems. Water Resour. Res. 43, 1-11. https://doi.org/10.1029/2006WR005636.

Lindbom, H., Tehler, H., Eriksson, K., Aven, T., 2015. The capability concept - On how to define and describe capability in relation to risk, vulnerability and resilience. Reliab. Eng. Syst. Saf. 135, 45-54. https://doi.org/10.1016/j.ress.2014.11.007.

Liu, X., Fang, Y.P., Ferrario, E., Zio, E., 2021. Resilience Measure and Importance Measure for Interdependent Critical Infrastructures. ASCE-ASME J. Risk Uncertain. Eng. Syst. Part B. Mech. Eng. 7 https://doi.org/10.1115/1.4051196.

MacEdo-Rouet, M., Rouet, J.F., Ros, C., Vibert, N., 2012. How do scientists select articles in the PubMed database? An empirical study of criteria and strategies. Rev. Eur. Psychol. Appl. 62, 63-72. https://doi.org/10.1016/j.eapr.2012.01.003.

Mackenzie, C.A., Barker, K., 2013. Empirical Data and Regression Analysis for Estimation of Infrastructure Resilience with Application to Electric Power Outages 19, 25-35. https://doi.org/10.1061/(ASCE)JS.1943-555X.0000103.

Madni, A.M., Jackson, S., 2009. Towards a conceptual framework for resilience engineering. IEEE Systems Journal 3 (2), 181-191.

Makadok, R., 2001. Toward a synthesis of the resource-based and dynamic-capability views of rent creation. Strategic management journal 22 (5), 387-401.

Manyena, S.B., 2006. The concept of resilience revisited. Disasters 30, 434-450. https:// doi.org/10.1111/j.0361-3666.2006.00031.x.

Markolf, S.A., Chester, M.V., Eisenberg, D.A., Iwaniec, D.M., Davidson, C.I., Zimmerman, R., Chang, H., 2018. Interdependent infrastructure as linked social, ecological, and technological systems (SETSs) to address lock-in and enhance resilience. Earth's Future 6 (12), 1638-1659.

Marples, D. R., 2006. The Political Consequences of the Chernobyl Disaster in Belarus and Ukraine. Chernobyl: The Event and its Aftermath.

Martin-Breen, P., Anderies, J. M., 2011. Resilience: A literature review. Bellagio Initiative, Brighton:IDS.
Meerow, S., Newell, J.P., Stults, M., 2016. Defining urban resilience: A review. Landsc. Urban Plan. 147, 38-49. https://doi.org/10.1016/j.landurbplan.2015.11.011.

Miller, L. E., Kelleher, J. J., Wong, L., 2004. Evaluation of network reliability calculation methods. *Assessment of Network Reliability Calculation Methods, JS Lee Associates, Inc.*

Miller-Hooks, E., Zhang, X., Futurechi, R., 2012. Measuring and maximizing resilience of freight transportation networks. Comput. Oper. Res. 39, 1633-1643. https://doi.org/10.1016/j.cor.2011.09.017.

Molyneaux, L., Brown, C., Wagner, L., Foster, J., 2016. Measuring resilience in energy systems: Insights from a range of disciplines. Renewable and Sustainable Energy Reviews 59, 1068-1079.

Morlok, E.K., Chang, D.J., 2004. Measuring capacity flexibility of a transportation system. Transp. Res. Part A Policy Pract. 38, 405-420. https://doi.org/10.1016/j.tra.2004.03.001.

Moteff, J., Copeland, C., & Fischer, J. (2003, January). Critical infrastructures: What makes an infrastructure critical?. Library of Congress Washington DC Congressional Research Service.

Mottahedi, A., Sereshki, F., Ataei, M., Nouri Qarahanlou, A., Barabadi, A., 2021. The resilience of critical infrastructure systems: a systematic literature review. Energies 14 (6), 1571.

Muller, G., 2012. Fuzzy architecture assessment for critical infrastructure resilience. Procedia Computer Science 12, 367-372.

Murdock, H.J., de Brujin, K.M., Gersonius, B., 2018. Assessment of critical infrastructure resilience to flooding using a response curve approach. Sustain. 10 https://doi.org/10.3390/su10103470.

Murino, G., Armando, A., Tacchella, A., 2019. Resilience of Cyber-Physical Systems: An Experimental Appraisal of Quantitative Measures. Int. Conf. Cyber Conflict, CYCON 2019-May, 1-19. https://doi.org/10.23919/CYCON.2019.8757010.

Najjar, W., Gaudiot, J.L., 1990. Network resilience: A measure of network fault tolerance. IEEE Transactions on Computers 39 (2), 174-181.

Nan, C., Sansavini, G., 2017. A quantitative method for assessing resilience of interdependent infrastructures. Reliability Engineering & System Safety 157, 35-53.

Omer, M., Nilchiani, R., Mostashari, A., 2009. Measuring the resilience of the global internet infrastructure system. 2009 IEEE Int. Syst. Conf. Proc. 156-162. https://doi.org/10.1109/SYSTEMS.2009.4815790.

Ouyang, M., Duenas-Osorio, L., 2012-b. Time-dependent resilience assessment and improvement of urban infrastructure systems. Chaos 22. https://doi.org/10.1063/1.4737204.

Ouyang, M., Duenas-Osorio, L., 2014. Multi-dimensional hurricane resilience assessment of electric power systems. Struct. Saf. 48, 15-24. https://doi.org/10.1016/j.strusafe.2014.01.001.

Ouyang, M., Duenas-Osorio, L., Min, X., 2012-a. A three-stage resilience analysis framework for urban infrastructure systems. Struct. Saf. 36-37, 23-31. https://doi.org/10.1016/j.strusafe.2011.12.004.

Ouyang, M., Wang, Z., 2015. Resilience assessment of interdependent infrastructure systems: With a focus on joint restoration modeling and analysis. Reliab. Eng. Syst. Saf. 141, 74-82. https://doi.org/10.1016/j.ress.2015.03.011.

Pant, R., Barker, K., Ramirez-Marquez, J.E., Rocco, C.M., 2014. Stochastic measures of resilience and their application to container terminals. Comput. Ind. Eng. 70, 183-194. https://doi.org/10.1016/j.cie.2014.01.017.

Peris-Mora, E., Orejas, J.D., Subirats, A., Ibáñez, S., Alvarez, P., 2005. Development of a system of indicators for sustainable port management. Marine pollution bulletin 50 (12), 1649-1660.

Petit, F. D. P., Bassett, G. W., Black, R., Buehling, W. A., Collins, M. J., Dickinson, D. C., ... Peerenboom, J. P., 2013. *Resilience measurement index: An indicator of critical infrastructure resilience* (No. ANL/DIS-13-01). Argonne National Lab.(ANL), Argonne, IL (United States).

Pietrucha-Urbaniak, K., Tchórzewska-Cieślak, B., Eid, M., 2020. Water network-failure data assessment. *Energies* 13 (11), 2990.

Pörtner, H. O., Roberts, D. C., Adams, H., Adler, C., Aldunce, P., Ali, E., Ara Begum, R., Betts, R., Beznér Kerr, R., Biesbroek, R., Birkmann, J., Bowen, K., Castellanos, E., Cissé, G., Constable, A., Cramer, W., Dodman, D., Eriksen, S. H., Fischlin, A., ... Zaiton Ibrahim, Z., 2022. Climate change 2022: impacts, adaptation and vulnerability. IPCC. https://edepot.wur.nl/565644.

Proag, V., 2014. The concept of vulnerability and resilience. Procedia Economics and Finance 18, 369-376.

Rania, N., Coppola, I., Martorana, F., Migliorini, L., 2019. The collapse of the Morandi Bridge in Genoa on 14 August 2018: A collective traumatic event and its emotional impact linked to the place and loss of a symbol. Sustainability 11 (23), 6822.

Reed, D.A., Kapur, K.C., Christie, R.D., 2009. Methodology for assessing the resilience of networked infrastructure. IEEE Syst. J. 3, 174-180. https://doi.org/10.1109/JSYST.2009.2017396.

Renn, O., 2008. White paper on risk governance: Toward an integrative framework. In Global risk governance (pp. 3-73). Springer, Dordrecht.

Rinaldi, S.M., Peerenboom, J.P., Kelly, T.K., 2001. Identifying, understanding, and analyzing critical infrastructure interdependencies. IEEE Control Syst. Mag. 21, 11-25. https://doi.org/10.1109/37.969131.

Robert, B., Hémond, Y., 2012. Organizational resilience: A multidisciplinary sociotechnical challenge. Resilience and urban risk management 119-125.

Roe, E., Schulman, P.R., 2012. Toward a comparative framework for measuring resilience in critical infrastructure systems. J. Comp. Policy Anal. Res. Pract. 14, 114-125. https://doi.org/10.1080/13876988.2012.664687.

Rosati, J.D., Touzinsky, K.F., 2015. Quantifying coastal system resilience for the US Army Corps of Engineers. Environ. Syst. Decis. 35, 196-208. https://doi.org/10.1007/s10669-015-9548-3.

Rose, A., Wei, D., 2013. Estimating the economic consequences of a port shutdown: the special role of resilience. Economic Systems Research 25 (2), 212-232.

Ruas, K., Kilar, V., Koren, D., 2018. Resilience assessment of complex urban systems to natural disasters: A new literature review. International journal of disaster risk reduction 31, 311-330.

Saurin, T.A., César, G., Júnior, C., 2011. Evaluation and improvement of a method for assessing HSMS from the resilience engineering perspective : A case study of an electricity distributor. Saf. Sci. 49, 355-368. https://doi.org/10.1016/j.ssci.2010.09.017.

Securité public Canada, 2022. https://www.securitepublique.gc.ca/cnt/rsrcs/pblctns/srctc-ctrl-nfrstrct/r/index-fr.aspx (accessed 27 September 2022).

Serre, D., 2018. Ds3 model testing: Assessing critical infrastructure network flood resilience at the neighbourhood scale. Urban B. Ser. 207-220 https://doi.org/10.1007/978-3-319-68606-6_13.

Serre, D., Heinzle, C., 2018. Assessing and mapping urban resilience to floods with respect to cascading effects through critical infrastructure networks. Int. J. Disaster Risk Reduct. 30, 235-243. https://doi.org/10.1016/j.ijdrr.2018.02.018.

Serre, D., 2016. Advanced methodology for risk and vulnerability assessment of interdependency of critical infrastructure in respect to urban floods. In E3S web of conferences (Vol. 7, p. 07002). EDP Sciences.

Shafieezadeh, A., Ivey Burden, L., 2014. Scenario-based resilience assessment framework for critical infrastructure systems: Case study for seismic resilience of seaports. Reliab. Eng. Syst. Saf. 132, 207-219. https://doi.org/10.1016/j.ress.2014.07.021.

Sharifi, A., Yamagata, Y., 2016. Urban resilience assessment: Multiple dimensions, criteria, and indicators. In Urban resilience. Springer, Cham, pp. 259-276.

Shavelson, R.J., McDonnell, L., Oakes, J., 1990. What are educational indicators and indicator systems? Practical Assessment, Research, and Evaluation 2 (1), 11.

Shavelson, R., 1987. *Indicator systems for monitoring mathematics and science education. Publications Department, The RAND Corporation, 1700 Main Street, Santa Monica, CA 90406-2138.*

Shirali, G.H.A., Mohammadfam, I., Motamedzade, M., Ebrahimpour, V., Moghimbeigi, A., 2012. Assessing resilience engineering based on safety culture and managerial factors. Process Safety Progress 31 (1), 17-18.

Shirali, G.A., Mohammadfam, I., Ebrahimpour, V., 2013. A new method for quantitative assessment of resilience engineering by PCA and NT approach: A case study in a process industry. Reliability Engineering & System Safety 119, 88-94.

Smith, A., Stirling, A., 2010. The politics of social-ecological resilience and sustainable socio-technical transitions. Ecology and society, 15(1). Engineering Resilience into Safety-Critical Systems.

Srinivasan, K. K., 2002. Transportation network vulnerability assessment: A quantitative framework. *Transportation Security Papers 2002.*

Spiegler, V.L.M., Naim, M.M., Wikner, J., n.d. n.d. 2012. A control engineering approach to the assessment of supply chain resilience. International journal of production research 50 (21), 6162-6187.

Sun, W., Bocchini, P., Davison, B.D., 2020. Resilience metrics and measurement methods for transportation infrastructure: the state of the art. Sustain. Resilient Infrastructure. 5, 168-199. https://doi.org/10.1080/23789689.2018.1448663.

Sun, Y., Turnquist, M.A., Nozick, L.K., 2006. Estimating freight transportation system capacity, flexibility, and degraded-condition performance. Transportation research record 1966 (1), 80-87.

Sun, Y., Zhang, Y., Gwizdka, J., Trace, C.B., 2019. Consumer evaluation of the quality of online health information: Systematic literature review of relevant criteria and indicators. J. Med. Internet Res. 21, 1-22. https://doi.org/10.2196/12522.

Tachaudomdach, S., Upayokin, A., Kronprasert, N., Arunotayanun, K., 2021. Quantifying road-network robustness toward flood-resilient transportation systems. Sustain. 13, 1-16. https://doi.org/10.3390/su13063172.

Tang, J., 2019. Assessment of Resilience in Complex Urban Systems. In: Leal, F.W., Azul, A., Brandli, L., Özuyar, P., Wall, T. (Eds.), Industry, Innovation and Infrastructure. Encyclopedia of the UN Sustainable Development Goals. Springer, Cham. https://doi.org/10.1007/978-3-319-71059-4_71-1.

Tatano, H., Tsuchiya, S., 2008. A framework for economic loss estimation due to seismic transportation network disruption: a spatial computable general equilibrium approach. Natural Hazards 44 (2), 253-265.

Tendall, D.M., Joerin, J., Kopainsky, B., Edwards, P., Shreck, A., Le, Q.B., Kruetli, P., Grant, M., Six, J., 2015. Food system resilience: Defining the concept. Glob. Food Sec. 6, 17-23. https://doi.org/10.1016/j.gfs.2015.08.001.

Tillement, S., Cholez, C., Reverdy, T., 2009. Assessing organizational resilience: an interactionist approach. Management 12 (4), 230-264.

Turnquist, M., Yugrin, E., 2013. Design for resilience in infrastructure distribution networks. Environment Systems & Decisions 33 (1), 104-120.

U.S Department of Homeland and Security, 2022. Critical Infrastructure Security & Resilience Research (CISR) Fact Sheet, https://www.dhs.gov/science-and-technology/publication/critical-infrastructure-security-resilience-research-fact-sheet (accessed 27 September 2022).

UNESCO-LearningPortal. https://learningportal.iiep.unesco.org/en (accessed 10 October 2022).

United Nations, 2016. World economic and social survey 2016. Climate change resilience: an opportunity for reducing inequalities. United Nations Department of Economic and Social Affairs.

United Nations, 2020. UN Common Guidance On Helping Build Resilient Societies 48.

Upadhyaya, J.K., Biswas, N., Tam, E.K.L., 2018. Managing for Change: Integrating Functionality, Resiliency, and Sustainability for Stormwater Infrastructure Assessment. J. Infrastuct. Syst. 24, 04018007. https://doi.org/10.1061/(asc) is.1943-555x.0000420.

Vogel, J., Canada, P.G., 1997. The Future Direction of Social Indicator Research Author (s): Joachim Vogel Published by : Springer Stable URL : https://www.jstor.org/stable/27522277 THE FUTURE DIRECTION OF SOCIAL INDICATOR 42, 103-116.
Vugrin, E.D., Camphouse, R.C., 2011, a. Infrastructure resilience assessment through control design 7, 243-260.

Vugrin, E.D., Warren, D.E., Ehlen, M.A., 2011. A resilience assessment framework for infrastructure and economic systems: Quantitative and qualitative resilience analysis of petrochemical supply chains to a hurricane. Process Safety Progress 30 (3), 280-290.

Vugrin, E.D., Warren, D.E., Ehlen, M.A., Camphouse, R.C., 2010. A framework for assessing the resilience of infrastructure and economic systems. In: Sustainable and resilient critical infrastructure systems. Springer, Berlin, Heidelberg, pp. 77-116.

Wang, J. Z., W., Rhode-Barbarigos, L., Lu, X., Wang, J., Lin, Y., 2019. Literature review on modeling and simulation of energy infrastructures from a resilience perspective. Reliab. Eng. Syst. Saf. 183, 360-373. https://doi.org/10.1016/j.ress.2018.11.029.

Wang, J.W., Gao, F., Ip, W.H., 2010. Measurement of resilience and its application to enterprise information systems. Enterprise information systems 4 (2), 215-223.

Wied, M., Oehmen, J., Welo, T., 2020. Conceptualizing resilience in engineering systems: An analysis of the literature. Systems Engineering 23 (1), 3-13.

Winkler, J., Duenas-Osorio, L., Stein, R., Subramanian, D., 2010. Performance assessment of topologically diverse power systems subjected to hurricane events. Reliability Engineering & System Safety 95 (4), 323-336.

Woods, D.D., 2019. Essentials of resilience, revisited. Handbook on resilience of socio-technical systems 52-65.

Xu, M., 2009. Evaluating sustainability and resilience of complex systems using network theory. Arizona State University.

Yang, Z., Barroca, B., Bony-Dandrieux, A., Dolidon, H., 2022-a.. Resilience Indicator of Urban Transport Infrastructure. A Review on Current Approaches. Infrastructures 7. https://doi.org/10.3390/infrastructures7030033.

Yang, Z., Clemente, M.F., Laffréchine, K., Heinzlef, C., Serre, D., Barroca, B., 2022-b.. Resilience of Social-Infrastructural Systems: Functional Interdependencies Analysis. Sustain. 14 https://doi.org/10.3390/su14020606.

Zhan, X., Meng, F., Liu, S., Fu, G., 2020. Comparing Performance Indicators for Assessing and Building Resilient Water Distribution Systems. J. Water Resour. Plan. Manag. 146, 06020012. https://doi.org/10.1061/(asce)wr.1943-5452.0001303.

Zhang, X., Miller-Hooks, E., Denny, K., 2015. Assessing the role of network topology in transportation network resilience. Journal of Transport Geography 46, 35-45.

Zhang, C., Ramirez-Marquez, J.E., Sanseverino, C.M.R., 2011. A holistic method for reliability performance assessment and critical components detection in complex networks. IIE Transactions 43 (9), 661-675.

Zhang, W., Wang, N., 2016. Resilience-based risk mitigation for road networks. Structural Safety 62, 57-65.