July 2015

NASA/TM-2015-218777 

Considering Risk and Resilience in DecisionMaking 

Wilfredo Torres-Pomales 

Langley Research Center, Hampton, Virginia 
NASA STI Program . . . in Profile

Since its founding, NASA has been dedicated to the 

advancement of aeronautics and space science. The 

NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role. 

The NASA STI program operates under the auspices 

of the Agency Chief Information Officer. It collects, 

organizes, provides for archiving, and disseminates 

NASA’s STI. The NASA STI program provides access 

to the NTRS Registered and its public interface, the 

NASA Technical Reports Server, thus providing one 

of the largest collections of aeronautical and space 

science STI in the world. Results are published in both 

non-NASA channels and by NASA in the NASA STI 

Report Series, which includes the following report 

types: 

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase of 

research that present the results of NASA 

Programs and include extensive data or theoretical 

analysis. Includes compilations of significant 

scientific and technical data and information 

deemed to be of continuing reference value. 

NASA counter-part of peer-reviewed formal 

professional papers but has less stringent 

limitations on manuscript length and extent of 

graphic presentations. 

• TECHNICAL MEMORANDUM. 

Scientific and technical findings that are 

preliminary or of specialized interest, 

e.g., quick release reports, working 

papers, and bibliographies that contain minimal 

annotation. Does not contain extensive analysis. 

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

• CONFERENCE PUBLICATION. 

Collected papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or 

co-sponsored by NASA. 

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from NASA 

programs, projects, and missions, often 

concerned with subjects having substantial 

public interest. 

• TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material pertinent to 

NASA’s mission. 

Specialized services also include organizing 

and publishing research results, distributing 

specialized research announcements and feeds, 

providing information desk and personal search 

support, and enabling data exchange services. 

For more information about the NASA STI program, 

see the following: 

• Access the NASA STI program home page at 

http://www.sti.nasa.gov 

• E-mail your question to help@sti.nasa.gov 

• Phone the NASA STI Information Desk at 

757-864-9658 

• Write to: 

NASA STI Information Desk 

Mail Stop 148 

NASA Langley Research Center 

Hampton, VA 23681-2199 
National Aeronautics and 

Space Administration 

Langley Research Center 

Hampton, Virginia 23681-2199 

 

July 2015

NASA/TM-2015-218777 

Considering Risk and Resilience in DecisionMaking 

Wilfredo Torres-Pomales 

Langley Research Center, Hampton, Virginia 
Available from: 

NASA STI Program / Mail Stop 148 

NASA Langley Research Center 

Hampton, VA 23681-2199 

Fax: 757-864-6500 

Acknowledgments 

The author is grateful for the comments and suggestions by Dr. Kenneth W. Eure and 

Yamira Santiago-Espada of NASA Langley Research Center. 
iii

Abstract 

This paper examines the concepts of decision-making, risk 

analysis, uncertainty and resilience analysis. The relation 

between risk, vulnerability, and resilience is analyzed. The paper 

describes how complexity, uncertainty, and ambiguity are the 

most critical factors in the definition of the approach and criteria 

for decision-making. Uncertainty in its various forms is what 

limits our ability to offer definitive answers to questions about the 

outcomes of alternatives in a decision-making process. It is shown 

that, although resilience-informed decision-making would seem 

fundamentally different from risk-informed decision-making, this 

is not the case as resilience-analysis can be easily incorporated 

within existing analytic-deliberative decision-making 

frameworks. 
iv

Table of Contents 

1. Introduction ................................................................................................................................................ 1

2. Decision-Making ........................................................................................................................................ 1

3. Risk Analysis .............................................................................................................................................. 3

4. Risk-Informed Decision-Making ................................................................................................................ 5

5. From Risk to Resilience ............................................................................................................................. 7

6. Resilience-Informed Decision-Making ...................................................................................................... 8

7. Final Remarks ............................................................................................................................................. 9

8. Bibliography ..............................................................................................................................................10
1

1. Introduction 

Decision-making is the process of choosing among alternatives. The nature and context of decisions 

can vary and there can be a variety of criteria to be considered in making a choice. The alternatives can 

involve a large number and different levels of uncertainties about the factors that influence the ultimate 

outcomes. For high-consequence decisions with potential outcomes that can have a large impact on valued 

things, such as human life, property, wealth, and the environment, the decision-making process should 

involve consideration of uncertainties about the outcomes. 

This paper presents an examination of the high-stakes decision-making process with consideration of 

the uncertainties associated with the alternatives. The next section presents a brief overview of the concept 

of decision-making. This is followed by a review of risk-informed decision-making, in which potential 

negative outcomes are considered in the selection of an alternative. In situations of high consequence and 

uncertainty, the robustness of the alternatives in the sense of insensitivity to uncertainties may be an 

advantageous consideration to increase confidence in the expected outcome of the selected alternative. A 

simple framework for resilience-informed decision-making is proposed to complement a risk-informed 

decision-making process. A summary and final remarks conclude this article. 

2. Decision-Making 

A decision is a resolution on an issue or a problem of interest being considered. The process of making 

a decision involves three elements: values, alternatives, and facts (Buede, 2009). Values are the criteria 

used to assess the utility of the outcome. Depending on the context, values can be objective or subjective, 

and quantitative or qualitative. The values capture the needs, objectives, and preferences of the 

stakeholders, which are the people and organizations that have an interest and can influence or be influenced 

by a decision. The alternatives may be given or can be the result of a creative process to generate potential 

solutions to the problem of interest. The facts are everything known about the alternatives and the context 

or environment in which the alternatives will be deployed or applied. Facts are the relevant data, 

information, and knowledge used in the process of assessing the alternatives against the values in order to 

make the decision. The decision makers use available facts, history, experience, and judgment to select an 

alternative. 

NASA’s Systems Engineering Handbook (NASA, 2007) describes a generic decision-making process. 

The essential steps are as follows: 

1. Define the criteria for evaluating alternative solutions 

2. Identify alternative solutions to address decision issues 

3. Select evaluation methods and tools 

4. Evaluate alternative solutions with the established criteria and selected methods 

5. Select recommended solutions from the alternatives based on the evaluation criteria 

6. Report analysis results with recommendations, impacts, and corrective actions 

In general, decision-making is an iterative analytic-deliberative process consisting of a technical subprocess to analyze the problem and the alternatives, and a deliberation sub-process to assess the alternatives 

and inform the analysis. The analysis and deliberation processes are iterated with information flowing 
2

between them until a final decision is made (Ersdal & Aven, 2008) (Zio & Pedroni, 2012). 

An important consideration in the definition of a decision-making process is the cost and time required 

to implement the process. Generally, a formal decision-making process is used only for decisions 

involving: 

 High stakes (i.e., the outcome can have a significant impact on cost, safety, etc.), 

 Complexity (i.e., outcomes are difficult to understand), 

 Uncertainty (i.e., inadequate knowledge to assess and rank alternatives with high confidence in the 

outcome), 

 Multiple criteria (i.e., formal analysis methods may be needed to resolve conflicts among 

evaluation criteria), and 

 Diversity of stakeholders (i.e., stakeholders may have different and conflicting values and 

preferences) (NASA, 2007) (Klinke & Renn, 2002). 

A formal decision-making process must be justified by a cost-benefit argument that considers the potential 

impact of the outcome and the resources and time needed to implement the process. 

Hester et al. (2012) have proposed a system for classifying stakeholders and determining an appropriate 

level of engagement in making decisions about a system. The classification attributes include: 

 Power (i.e., ability to influence other stakeholders), 

 Legitimacy (i.e., actions are proper or desirable within some socially constructed system of norms, 

beliefs, or definitions), and 

 Urgency (i.e., degree to which stakeholder’s views require immediate action). 

Additionally, Hester et al. consider the level of support of particular stakeholders for system-related 

decisions, with the categories of supportive, mixed, non-supportive, and marginal. Based on the patterns 

of degrees of these attributes, Hester et al. propose a set of strategies for engagement and involvement of 

the stakeholders in the decision-making process. These strategies include: involve, collaborate, defend, 

monitor, and no action. 

A number of techniques are available for individual and group decision-making. Techniques for 

individual decision-making include: 

 Prioritization (i.e., rank by order of importance), 

 Satisficing (i.e., heuristic of selecting the first alternative that meets minimum requirements), 

 Opportunity cost (i.e., the benefit received from selecting a different alternative), 

 Bureaucratic (i.e., structured and objective criteria-based), and 
3

 Elimination (i.e., process of comparison and elimination of alternatives that do not meet selected 

criteria and continuing until only one alternative remains). 

Group decision-making techniques include: 

 Consensus (i.e., majority decision with minority support to go along), 

 Voting (e.g., majority or plurality), and 

 Delphi method (Helmer, 1967). 

A major concern and source of difficulty in high-consequence decision-making is the presence of 

uncertainty about the alternatives. The causes of uncertainty include: 

 Lack of information or knowledge, 

 Abundance of information or knowledge, 

 Conflicting nature of pieces of information, 

 Measurement errors, linguistic ambiguity, and 

 Subjectivity of analysis opinions (Zio & Pedroni, 2012). 

Uncertainties can also be classified as: 

 Knowledge uncertainty (i.e., only sparse statistics or random data are available), 

 Modeling uncertainty (i.e., incomplete or inaccurate models), and 

 Limited predictability or unpredictability (i.e., outcome is highly sensitive to initial conditions) 

(Aven T. , 2006). 

Fundamentally, decision-making uncertainties are either aleatory (i.e., due to inherently random processes) 

or epistemic (i.e., due to lack of knowledge about properties and conditions of phenomena that determine 

the behavior of a system) (Zio & Pedroni, 2012). Epistemic uncertainty can be reduced by acquiring 

knowledge, but aleatory uncertainty is irreducible. Uncertainty is a fundamental limitation on the 

effectiveness of a decision-making process. 

3. Risk Analysis 

Risk has been defined as “the potential for an unwanted outcome resulting from an incident, event, or 

occurrence as determined by its likelihood and the associated consequences” (DHS, 2010). However, there 

are multiple perspectives on the concept of risk (Aven T. , 2006). One definition of risk is the combination

of probabilities and consequences, where the consequences are assessed in terms of valued things such as 

health, the environment, and safety. Another definition of risk is the expected value of consequences given 

by the product of probabilities and consequences. Risk has also been defined as a combination of the 
4

elements of events, consequences, probabilities, and uncertainties. In the most general approach, these 

elements are kept and handled separate, as there is no agreement on a reducing mathematical formulation 

that is universally suitable for all decision situations (Aven T. , 2010). Other definitions of risk include the 

“effect of uncertainty on objectives” and “a situation or event where something of human value (including 

humans themselves) has been put at stake and where outcome is uncertain” (Aven T. , 2010). 

There are seven domains where risk is a major consideration: 

 Environmental risk (i.e., related to events such as floods and pollution), 

 Lifestyle risk (i.e., related to food, drugs, driving, and so on), 

 Medical risk (i.e., related to medical treatments, surgery, etc.), 

 Interpersonal risk (i.e., social relationships, parenting, and others), 

 Economic risk (i.e., such as investment, employment decisions and events), 

 Criminal risk (i.e., being a victim of illegal activity), and 

 Technical risk (i.e., accidental side effects of technological innovations) (Johansen, 2010). 

In all domains, the focus is on likelihood and severity of future consequences of present decisions and 

actions. 

One view on the purpose of risk analysis is “to understand a risk in order to do something about it” 

(Johansen, 2010). However, at a more basic level, risk reduction is not the primary goal of a risk 

assessment. Risk analysis serves to provide input to a particular decision-making process with a wider 

range of additional considerations (Johansen, 2010). Thus, risk assessment is a tool to generate information 

that contributes to reducing the uncertainty of decision makers about the outcome of decision alternatives. 

Risk analysis can be based on deterministic and probabilistic analysis methods (Kirchsteiger, 1999). 

Deterministic risk analysis is based on worst-case assumptions that bound the uncertainties on likelihood 

and severity of consequences. This effectively simplifies the analysis and provides reasonable assurance 

that the risk level is acceptable. A problem with this approach is that the definition of worst-case conditions 

is subjective and potentially arbitrary, which can lead to consideration of scenarios which are possible but 

highly unlikely. Deterministic risk analysis is a conservative approach that offers only broad (and not deep) 

insight into the likelihood of possible outcomes. Probabilistic risk analysis (PRA) is intended to be more 

objective, detailed, and quantitative and offer deeper insight into the relative likelihood of outcomes and 

the implications of uncertainty. NASA has published a comprehensive guide on PRAs (NASA, 2011). 

PRA is centered on the quantification of risk metrics, which are probabilistic performance measures that 

appear in decision models. Examples of risk metrics are the probabilities of fuel tank explosion and of 

system failure. Such metrics and the techniques used in their calculations can offer insight into the relative 

importance of various sources of uncertainty and serve as a reference in optimal resource allocation 

decisions. In general, the output of a risk analysis consists of undesirable risk-significant scenarios, the 

consequences of those scenarios, the probabilities, and the uncertainty or confidence in the results of the 

analysis (NASA, 2011). 

Various processes have been proposed for risk analysis (NASA, 2010) (NASA, 2011) (Johansen, 2010) 
5

(Biringer, Vugrin, & Warren, 2013). The basic steps are: 

1. Define the system, 

2. Identify the alternatives, 

3. Identify the hazards (i.e., what can go wrong), 

4. Estimate the likelihood and consequences for each hazard, and 

5. Combine the results into a single comprehensive risk picture. 

A wide range of analysis tools and techniques are available, including fault tree analysis; failure mode and 

effects analysis; and modeling and simulation. The risk analysis results are used in risk assessments where 

the goal is to assess the confidence in the analysis results and decide whether the level of risk is acceptable, 

tolerable (i.e., operationally manageable), or unacceptable. 

The effectiveness of risk assessments is complicated by three major factors that limit the accuracy of 

the results (Ben-Haim, 2012). These include our evolving understanding of the world and its uncertainties, 

the inherent random nature of physical phenomena, and the fact that learning from experience involves 

untestable assumptions in rational processes of inductive reasoning about the physical world. 

Considerations of ontological uncertainties (i.e., what exists and its nature) and epistemological 

uncertainties (i.e., acquisition and thoroughness of knowledge about what exists) complicate the assessment 

of risk as it is clear that truly objective and accurate assessment of risk is not possible, except for the simplest 

of situations. Situations involving complexity, uncertainty, and ambiguity necessarily require subjective 

judgments and considerations of stakeholder values and preferences to arrive at a decision. 

4. Risk-Informed Decision-Making 

There are two basic approaches for using risk information in decision-making: risk-based and riskinformed (Johansen, 2010). Risk-based decision-making considers only risk assessment results to make 

decisions. This approach may be adequate for simple systems when the stakeholders are aligned in their 

values and preferences. Safety-critical decision-making processes may follow this approach. On the other 

hand, risk-informed decision-making (RIDM) considers risk along with other sources of information, 

such as cost-benefit analyses and laws and regulations, to compare and deliberate on the preferred 

alternative. The basic steps of RIDM are: 

1. Selection of alternatives, 

2. Risk analysis for each alternative, and 

3. Alternative selection (Zio & Pedroni, 2012) (NASA, 2010). 

Note that RIDM is an iterative analytic and deliberative process in which information can flow both ways 

between analysis and deliberation to ensure that alternative generation, risk analysis, and deliberation for 

alternative selection are adequate and as detailed and complete as needed to ensure an explicit and 

defensible decision. RIDM is the preferred approach when there is a multitude of stakeholder values and 
6

preferences (i.e., ambiguity), and there is a need to find the best alternative that satisfies multiple objective 

and subjective criteria. Both deterministic and probabilistic analysis results may be use in RIDM. Large 

groups of stakeholders from diverse segments of society may participate in the generation of alternatives 

and the process of deliberation (NRC, 1996). In such situations, risks may be traded against other attributes 

in order to ensure selection of the best overall alternative (NASA, 2007) (Amendolla, 2001). 

The research literature on risk analysis and risk-informed decision-making is extensive. The principal 

issues are the disambiguation and definition of concepts, dealing with uncertainty, strategies for dealing 

with risk, and communicating risk to the decision makers1

. Risk and RIDM are very complex topics and 

there is not a single solution that fits all possible situations. Instead, each situation requires consideration 

of the specific circumstances and framing the problems and approach to ensure adequate coverage of 

relevant factors. Complexity, uncertainty, and ambiguity are the major categories of consideration in 

choosing a RIDM approach (Amendolla, 2001). 

Ersdal and Aven (2008) have identified five common decision principles used in RIDM: 

 Cost-benefit analysis (i.e., choose the alternative that maximizes the total benefit taking into account 

costs, possible losses due to negative outcomes, and any expected income or benefit); 

 Risk acceptance criteria (i.e., the risk is compared to a threshold level of acceptability); 

 Cautionary and precautionary principle (i.e., seek alternatives that are robust or insensitive to 

uncertainties in the likelihood of severe outcomes by avoiding those outcomes and reducing their 

impact should the undesired outcomes occur); 

 As Low As Reasonably Practicable (ALARP) principle (i.e., all available risk reduction (likelihood, 

severity, and uncertainty) should be implemented unless gross disproportion between cost and benefit 

can be demonstrated); and 

 Multi-attribute analysis (i.e., where multiple criteria, including risk-related ethical factors, must be 

considered in deliberations to identify a suitable and balanced alternative). 

Multi-attribute decision-making is the most general situation. The decision principle applied depends on 

ethical, safety, and economic considerations, as well as the risk posture of the stakeholders (i.e., the 

acceptable level of risk for a particular potential level of reward). 

Decisions with high uncertainty and potentially high negative consequences require special 

consideration. In these situations, it may be the judgment of experts and scientists that the likelihood of 

strongly negative consequences is not insignificant, but the effectiveness of a risk analysis is limited by a 

high level of epistemic and aleatory uncertainty. Ethical considerations may then be strong determinants 

of the proper course of action (Aven T. , 2006). In some situations, the major uncertainty is in the likelihood 

of the hazards. This is the case in infrastructure risk assessments where the threats can be random or 

deliberate with the range of threats including natural events, accidents, and malevolent actions (Biringer, 

Vugrin, & Warren, 2013). In this case, it may be advantageous to avoid direct estimation of the likelihood 

of hazards and instead consider assessments of the overall threat environment (i.e., possibilities, 

capabilities, motive, etc.) (Haimes, 2009). Application of the cautionary and precautionary principle then 

leads to consideration of alternatives aimed at avoiding hazards and mitigating the effects of hazards that 

 

1

 The Bibliography lists many relevant references on these issues. 
7

may occur (Johansen, 2010). In effect, the desirable alternative is one that offers a high degree of robustness 

(i.e., insensitivity), and thus, assurance against uncertainties. 

5. From Risk to Resilience 

The concept of risk is not adequate if the likelihood of hazards is highly uncertain. In that case, the 

emphasis should be on the analysis of the consequences of potential hazards. For this, risk is decomposed 

into hazards and vulnerability (Ciurean, Schroter, & Glade, 2013): 

Risk = Hazard x Vulnerability 

A hazard is a “dangerous phenomenon, substance, human activity, or condition that may cause loss of 

life, injury or other health impacts, property damage, loss of livelihoods and services, social and economic 

disruption, or environmental damage” (Ciurean, Schroter, & Glade, 2013). In addition, Ciurean et al. 

(2013) state that “Within the risk management framework, vulnerability pertains to consequence analysis” 

and “defines the potential for loss to the elements at risk caused by the occurrence of a hazard.” 

Furthermore, Aven (2011) has referred to vulnerability as the antonym of robustness. According to 

Johansen (2010): “Regardless of the improbability of an event, unfortunate outcomes may still occur. 

Reducing vulnerability on the other hand, will always attenuate risk through reduced consequence severity. 

Especially important in this are cases of great epistemic uncertainty.” Johansen describes vulnerability as 

composed of three elements: 

 Exposure (i.e., effectiveness of protection against initiation of disruption due to hazards), 

 Sensitivity (i.e., degradation in performance due to propagation of hazard effects), and 

 Resilience (i.e., ability to bounce back after a disturbance). 

Robust mitigation of hazard effects requires a balanced allocation of resources to minimize vulnerability in 

a cost-effective manner. All of these factors would be considered in a comprehensive analytic-deliberative 

decision-making process for high consequence situation. 

Francis and Bekera (2014) proposed a resilience concept that encompasses three capacities: 

 Absorptive (i.e., ability to absorb and contain the effects of hazards), 

 Adaptive (i.e., the ability to internally adapt to compensate for exceedance of absorptive capacity), 

and 

 Recovery/restorative (i.e., ability to return to normal or improved level of performance). 

Many other definitions of resilience have been proposed in the literature. Biringer et al. (2013) list 23 

different definitions proposed by different authors. Haimes (2009) defined resilience as “the ability of the 

system to withstand a major disruption within acceptable degradation parameters and to recover within 

acceptable time and composite costs and risks.” Aven (2011) has also examined some existing definitions 

of vulnerability and resilience in the literature and proposed that vulnerability is assessed with respect to a 

fixed set of hazard events, while resilience is assessed relative to any set of hazards. From this perspective, 
8

vulnerability is a special case of resilience. Biringer et al.’s decomposition seems better aligned with the 

common sense of the notion of resilience as the ability to cope with change, including resisting degradation 

and recovering in a timely manner. From this perspective, Biringer et al.’s concept of protection aligns 

with Johansen’s concept of exposure, and Biringer et al.’s resilience would include the concepts of 

sensitivity and resilience in the sense described by Johansen (2010). That is: 

Vulnerability = Protection x Resilience 

and 

Risk = Hazard x Protection x Resilience. 

6. Resilience-Informed Decision-Making 

Risk analysis considers three questions: What can go wrong? How likely is it? What are the associated 

consequences? (NASA, 2011). A risk analysis should also consider the uncertainty. Thus, the risk analysis 

elements are: 

 Scenarios, 

 Likelihood, 

 Consequences, and 

 Uncertainty. 

From this perspective, resilience analysis should be an explicit and integral element of the analysis of 

consequences, and the results of these analyses should be included in the considerations of risk-informed 

decision-making. Alternatively, a dedicated resilience analysis would consider a wide range of potential 

hazards to assess the capability of the system to absorb, adapt, and recover. Resilience analysis would be 

similar to robustness and sensitivity analyses and have the goal of determining the range of possible effects 

under conditions of high uncertainty. 

A resilience-informed decision-making process would not be significantly different from a riskinformed decision-making process. Both approaches acknowledge that decision-making for high 

consequence situations requires consideration of technical as well as social factors such as the values and 

preferences of the stakeholders, especially to define the decision criteria and the acceptable level of 

uncertainty in the outcome of decisions. 

Figure 1 illustrates a generic informed decision-making framework that includes various types of 

analyses to generate critical information for consideration in the process of stakeholder deliberation. This 

framework is based on the one presented by Johansen (2010), and it is similar to the Risk-Informed 

Decision-Framework used by NASA and the US Nuclear Regulatory Commission (USNRC) (Zio & 

Pedroni, 2012) (NASA, 2010) (NASA, 2011) (NASA, 2007) (Johansen, 2010). 
9

7. Final Remarks 

This paper has presented an examination of the concepts of decision-making, risk analysis, uncertainty, 

and resilience analysis. The relation between risk, vulnerability, and resilience was analyzed. In addition, 

a resilience-informed decision-making approach that builds on existing informed decision-making 

frameworks was proposed. 

This paper provides insight into the breadth and depth of available published work produced by risk 

researchers worldwide. Additionally, the paper shows that, although resilience-informed decision-making 

would seem fundamentally different from risk-based decision-making, this is not the case as resilienceanalysis can be easily incorporated within existing analytic-deliberative decision-making frameworks. The 

paper also describes how complexity, uncertainty, and ambiguity are the most critical factors in the 

definition of an approach and criteria for decision-making. The paper also offers insight into the approaches 

for structuring, analyzing, and managing uncertainty. Ultimately, uncertainty in its various forms is what 

limits our ability to offer definitive answers to questions about the outcomes of alternatives in a decisionmaking process. 

Laws, Regulations, Stakeholder 

Values and Preferences 

Analysis of Alternatives

 Cost-Benefit 

 Risk 

 Resilience 

 Others

Problem, 

Issue Decision 

Review, 

Deliberation, 

and Alternative 

Selection 

Generation of 

Alternatives 

Figure 1: General Analytic-Deliberative Decision-Making Framework (adapted from (Johansen, 2010))
10

8. Bibliography 

Amendolla, A. (2001). Recent paradigms for risk informed decision making. Safety Science, 40, 

17 - 30.

Aven, T. (2006, July). On the Precautionary Principle, in the Context of Different Perspectives of 

Risk. Risk Management, 8(3), 192 - 205. 

Aven, T. (2010). On how to define, understand and describe risk. Reliability Engineering and 

System Safety, 95, 623 - 631. 

Aven, T. (2011). On Some Recent Definitions and Analysis Frameworks for Risk, Vulnerability, 

and Resilience. Risk Analysis, 31(4), 515 - 522. 

Aven, T. (2015). Implications of black swans to the foundations and practice of risk assessment 

and management. Reliability Engineering and System Safety, 134, 83 - 91. 

Aven, T., & Steen, R. (2010). The concept of ignorance in a risk assessment and risk 

management context. Reliability Engineering and System Safety, 95, 1117 - 1122. 

Aven, T., & Zio, E. (2011). Some considerations on the treatment of uncertainties in risk 

assessment for practical decision making. Reliability Engineering and System Safety, 96, 

64 - 74.

Ben-Haim, Y. (2012). Why Risk Analysis is Difficult, and Some Thoughts on How to Proceed. 

Risk Analysis, 32(10), 1638 - 1646. 

Biringer, B. E., Vugrin, E. D., & Warren, D. E. (2013). Critical Infrastructure System Security 

and Resilience. CRC Press. 

Buede, D. M. (2009). The Engineering Design of Systems: Models and Methods (2nd ed.). 

Wiley. 

Ciurean, R. L., Schroter, D., & Glade, T. (2013). Conceptual Frameworks of Vulnerability 

Assessments for Natural Disasters Reduction. In J. Tiefenbacher, Approaches to Disaster 

Management - Examining the Implications of Hazards, Emergencies and Disasters.

Retrieved from http://www.intechopen.com/books/approaches-to-disaster-management-
11

examining-the-implications-of-hazards-emergencies-and-disasters/conceptualframeworks-of-vulnerability-assessments-for-natural-disasters-reduction 

DHS. (2010). DHS Risk Lexicon. Department of Homeland Security. 

Ersdal, G., & Aven, T. (2008). Risk informed decision-making and its ethical basis. Reliability 

Engineering and System Safety, 93, 197–205. 

Francis, R., & Bekera, B. (2014). A metric and frameworks for resilience analysis of engineered 

and infrastructure systems. Reliability Engineering and System Safety, 121, 90 - 103.

Haimes, Y. Y. (2009). On the Definition of Resilience in Systems. Risk Analysis, 29(4), 498 - 

501.

Helmer, O. (1967). Analysis of the Future: The Delphi Method. RAND Corporation. Retrieved 

from http://www.rand.org/pubs/papers/P3558.html 

Henry, D., & Ramirez-Marquez, J. E. (2012). Generic metrics and quantitative approaches for 

system resilience as a function of time. Reliability Enginering and System Safety, 99, 114 

- 122.

Hester, P. T., Bradley, J. M., & Adams, K. M. (2012). Stakeholders in systems of systems. 

International Journal Systems of Systems Engineering, 3(3/4), 225 - 232. 

Homeland Security Studies & Analysis Institute. (2010). Risk and Resilience: Exploring the 

Relationship.

Johansen, I. L. (2010). Foundations of Risk Assessment. Norwegian University of Science and 

Technology (NTNU). 

Kirchsteiger, C. (1999). On the use of probabilistic and deterministic methods in risk analysis. 

Journal of Loss Prevention in the Process Industries, 12, 399–419.

Klinke, A., & Renn, O. (2002). A New Approach to Risk Evaluation and Management: RiskBased, Precaution-Based, and Discourse-Based Strategies. Risk Analysis, 22(6), 1071 - 

1094.

Krebs, J. R. (2011, December 13). Risk, Uncertainty and Regulation. Philosophical 

Transactions: Mathematical, Physical and Engineering Sciences, 369(1956), 4842-4852. 
12

Lempert, R. J., & Collins, M. T. (2007). Managing the Risk of Uncertain Threshold Responses: 

Comparison of Robust, Optimum, and Precautionary Aproaches. Risk Analysis, 27(4), 

1009 - 1026.

NASA. (2007). NASA Systems Engineering Handbook (NASA/SP-2007-6105). National 

Aeronautics and Space Administration. 

NASA. (2010). NASA Risk-Informed Decision Making Handbook (NASA/SP-2010-576).

National Aeronautics and Space Administration. 

NASA. (2011). NASA Risk Management Handbook (NASA/SP-2011-3422). National 

Aeronautics and Space Administration. 

NASA. (2011). Probabilistic Risk Assessment Procedures Guide for NASA Managers and 

Practitioners (NASA/SP-2011-3421). National Aeronautics and Space Administration. 

NRC. (1996). Understanding Risk: Informing Decisions in a Democratic Society. (P. C. Stern, & 

H. V. Fineberg, Eds.) Washington, D.C.: Committee on Risk Characterization, National 

Research Coucil. 

Pate-Cornell, E. (2012). On "Black Swans" and "Perfect Storms": Risk Analysis and 

Management when Statistics are Not Enough. Risk Analysis, 32(11), 1823 - 1833. 

Pate-Cornell, M. E. (1996). Uncertainties in risk analysis: Six levels of treatment. Reliability 

Enginering and System Safety, 54, 95 - 111. 

Steen, R., & Aven, T. (2011, Feburary). A risk perspective suitable for resilience engineering. 

Safety Science, 49(2), 292 - 297.

Zio, E., & Pedroni, N. (2012). Risk-informed decision-making processes: An overview.

Foundation for an Industrial Safety Culture (FonCSI). 

Zio, E., & Pedroni, N. (2012). Uncertainty characterization in risk analysis for decision-making 

practice. Foundation for an Industrial Safety Culture (FonCSI). 
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Technical Memorandum

 4. TITLE AND SUBTITLE

 Considering Risk and Resilience in Decision-Making 

5a. CONTRACT NUMBER

 6. AUTHOR(S)

Torres-Pomales, Wilfredo

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

 8. PERFORMING ORGANIZATION

 REPORT NUMBER

L-20564

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 31

Availability: NASA STI Program (757) 864-9658

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

This paper examines the concepts of decision-making, risk analysis, uncertainty and resilience analysis. The relation between 

risk, vulnerability, and resilience is analyzed. The paper describes how complexity, uncertainty, and ambiguity are the most 

critical factors in the definition of the approach and criteria for decision-making. Uncertainty in its various forms is what limits 

our ability to offer definitive answers to questions about the outcomes of alternatives in a decision-making process. It is shown 

that, although resilience-informed decision-making would seem fundamentally different from risk-informed decision-making, 

this is not the case as resilience-analysis can be easily incorporated within existing analytic-deliberative decision-making 

frameworks. 

15. SUBJECT TERMS

Complexity; Consequence; Likelihood; Resilience; Risk; Uncertainty

18. NUMBER

 OF 

 PAGES

19

19b. TELEPHONE NUMBER (Include area code)

(757) 864-9658

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

 999182.02.50.07.02 

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA-TM-2015-218777

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 07 - 2015