**Feasibility Study to Evaluate**

**Enterprise Risk Management Methodology**

August 7, 2006

Prepared for

National Institutes of Health

Office of the Director/Office of Management Assessment

Submitted by

William G. Foote, Ph.D.

Deloitte & Touche LLP

[Executive Summary 3](#executive-summary)

[NIH-wide Requirements 3](#nih-wide-requirements)

[Feasibility Study Design 3](#feasibility-study-design)

[Summary Observations of the Assessment
3](#summary-observations-of-the-assessment)

[Summary of Recommendations 3](#summary-of-recommendations)

[Summary of Answers to Study Questions
3](#summary-of-answers-to-study-questions)

[1. Introduction 3](#introduction)

[1.1 Overview of NIH-wide Risk Assessment
3](#overview-of-nih-wide-risk-assessment)

[1.2 Purpose 3](#purpose)

[1.3 ERM Applied to NIH 3](#erm-applied-to-nih)

[1.4 Background 3](#background)

[2. Study Design and Methodology 3](#study-design-and-methodology)

[2.1 Conceptual Framework of the Study
3](#conceptual-framework-of-the-study)

[2.2 Data Collection and Analysis Procedures
3](#data-collection-and-analysis-procedures)

[2.3 NIH Documents 3](#nih-documents)

[2.4 Open Source Searches 3](#open-source-searches)

[2.5 NIH Site Visits 3](#nih-site-visits)

[3. Assessment of ERM Methodologies 3](#assessment-of-erm-methodologies)

[3.1 Overview 3](#overview)

[3.2 Assessment Perspectives from the Social Sciences
3](#assessment-perspectives-from-the-social-sciences)

[Political Science and Organizational Behavior Observations
3](#political-science-and-organizational-behavior-observations)

[Observations from Decision Making Psychology
3](#observations-from-decision-making-psychology)

[Observations from Experimental Economics and Finance
3](#observations-from-experimental-economics-and-finance)

[3.3 Standards and Guidance for Risk Assessment
3](#standards-and-guidance-for-risk-assessment)

[3.4 Examples of Risk Assessment Methodologies
3](#examples-of-risk-assessment-methodologies)

[3.5 Preliminary Observations from an ERM Survey
3](#preliminary-observations-from-an-erm-survey)

[3.6 Case Examples of Recent ERA Development
3](#case-examples-of-recent-era-development)

[Case 1: Subsidiary of a Major Global Process Chemical Company
3](#case-1-subsidiary-of-a-major-global-process-chemical-company)

[Case 2: Global Marketer, Producer, and Distributor of Beverages
3](#case-2-global-marketer-producer-and-distributor-of-beverages)

[Case 3: State Retirement Agency 3](#case-3-state-retirement-agency)

[3.7 NIH specific Information 3](#nih-specific-information)

[4. Summary of Recommendations 3](#summary-of-recommendations-1)

[4.1 Recommendations Based on 8 Dimensions of Risk Intelligence
3](#recommendations-based-on-8-dimensions-of-risk-intelligence)

[4.2 Proposed Enterprise Risk Assessment Process
3](#proposed-enterprise-risk-assessment-process)

[4.3 Three Options 3](#three-options)

[4.3 Three Options 3](#three-options)

[Option A: Initiate Facilitated Assessment:
3](#option-a-initiate-facilitated-assessment)

[Option B: Initiate Data-Driven Facilitated Assessment
3](#option-b-initiate-data-driven-facilitated-assessment)

[Option C: Develop Comprehensive Data-Driven Assessment Environment
3](#option-c-develop-comprehensive-data-driven-assessment-environment)

[4.5 Study Questions and Summary Answers
3](#study-questions-and-summary-answers)

[5. Summary of References Consulted 3](#summary-of-references-consulted)

[APPENDICES 3](#appendices)

[Appendix A: Interviewees (status as of August 5, 2006)
3](#appendix-a-interviewees-status-as-of-august-5-2006)

[Appendix B: Documents Reviewed 3](#appendix-b-documents-reviewed)

[Appendix C: Summary High-Level Media Scan
3](#appendix-c-summary-high-level-media-scan)

[LIST OF SELECTED RISK IQ SITES 3](#list-of-selected-risk-iq-sites)

[Appendix D: Risk IQ 3](#appendix-d-risk-iq)

[Frequently Used Terms in this Study
3](#frequently-used-terms-in-this-study)

#   {#section .unnumbered}

# Executive Summary

## NIH-wide Requirements

The goal of this feasibility study ("study") is to provide a
scientifically sound[^1] design for an enterprise-wide risk assessment
("assessment") methodology.[^2] The study was prompted by National
Institutes of Health ("NIH") Director's interest in establishing sound
management leading practices in risk management across NIH. This
interest is supported by the GAO High Risks report of an emerging area
for all Federal agencies, "that instilling a disciplined approach to
identifying and managing risk has broad applicability across a wide
range of federal programs, operations, and functions across the federal
government. This will be a continuing focus of NIH's work in the
future.[^3]

For the purposes of this study, risk is "a possible loss due to failure
of NIH to perform against its mission." Failure can emanate from any
combination of factors, including those generated by people, process,
systems, technology, science, or occasioned by external events. The goal
of an enterprise risk assessment is to prioritize risks, segmented by
their location of occurrence (e.g., process and system, organizational
unit), in order to effectively and efficiently allocate resources to
prevent/mitigate specific risks.

This study provisionally answers three questions:

1.  What is the best methodology for the NIH to use to evaluate risk for
    its scientific, administrative, and financial programs for the
    NIH-wide risk assessment in Phase 2, and in the future?

2.  What is the most economical, efficient and effective way to collect
    the data needed to evaluate these high risk areas without imposing
    an excessive burden on program staff or NIH administrators?

3.  What is the baseline status of the existing risk areas previously
    identified by NIH managers?

The short answer to question one is that several methodologies exist and
have been used by various organizations. This study will systematically
assess the methodologies that have been used to ascertain a set of
practices useful for NIH to consider in assessing and managing
enterprise risk. The study will also consider approaches that offer
fewer burdens on the business of operating NIH programs. In the course
of the study several risks can be identified to suggest a baseline for
commencing an assessment of NIH risks to the performance of NIH mission.

This report describes the study design and conceptual framework for
assessment of NIH-wide risk, recommends a methodology that is feasible
and is not overly onerous for NIH executives and administrators. The
recommendations are suitable for any NIH office, center or institute as
well as NIH-wide risk assessment activities.

## Feasibility Study Design

The feasibility study for developing a leading practices NIH-wide risk
assessment methodology addresses the above three questions. Several
constraints make scientific study of risk management methods
challenging:

-   The real values for loss are not known, or may even be unknowable *a
    priori*.

-   Each set of events occurring to an organization is unique and not
    repeatable; and worse yet, ensembles of events are unpredictable.

-   Risk management methods cannot be separated from the object of
    study: if a method results in some action, the state of the
    organization that employs the method irrevocably changes.[^4]

-   Introduction of a risk management method to an organization changes
    the behavior of participants in an organization, as well as the way
    participants may deploy processes and systems in response to
    systematic knowledge about potential losses.

-   A single occurrence of a risk, whether predicted or not, cannot be
    used to draw any conclusions about the accuracy of risk analysis
    methods.

-   The objects of an NIH-wide risk assessment (strategies, processes,
    budgets, resources) have relatively long cycle times.

The above constraints may limit the empirical study design options
available in risk management to answer the three questions. These
constraints do not prevent an application of systematic, logical and
empirical scientific principles. Recognition of these constraints allows
a design to provide more reliable results than anecdotal descriptions of
risk assessment practices.

The organizing principle used to address the three questions is a *study
conceptual framework* that incorporates multiple approaches to data and
analysis. These approaches are used to converge on assessment
methodology practices based on the principle of relating similar
practices with one another. Thus the approach "triangulates" the
analysis to provide provisional conclusions regarding leading enterprise
risk assessment practices:

1.  Review of risk assessment decision making literature

2.  Existing risk assessment guidance

3.  Exploratory survey of risk assessment practices

4.  Case examples of recent enterprise risk assessment developments

5.  Interviews with NIH executives and review of NIH documents

Study questions, assumptions and constraints feed into baseline data
gathering from NIH documents, risk literature, and risk standards and
guidance. With this data the eight dimensions of a risk intelligent[^5]
organization are developed and used to inform NIH interviews, construct
case studies and interpret preliminary Enterprise Risk Management
("ERM") survey results. Interviews, document, literature and standards
review, case studies, and preliminary survey results then provide the
basis for answering the three study questions.

## Summary Observations of the Assessment

Presented in this section are the NIH ERM team's summary of high-level
observations of current ERM and other risk assessment practice and
recommendations for continued improvement of sustainable practice. The
summary of recommendations is organized according to the eight
dimensions of risk intelligence. These dimensions are consistent with
COSO (II) ERM, as well as a variety of risk management and risk-related
guidance, standards, and practices.

-   **Governance and Risk Oversight**-- Are NIH standards of behavior,
    values, integrity aligned with internal & external stakeholder
    expectations? Are NIH strategy and execution objectives aligned to
    operations, infrastructure, compliance and reporting objectives?

-   **Risk Identification** -- Does NIH management identify potential
    key scenarios and analyze the risks, opportunities and
    interdependencies of these?

-   **Risk Assessment** **and Measurement** -- Does NIH evaluate events
    from an impact and vulnerability perspective, using a combination of
    quantitative and qualitative techniques?

-   **Response Alternatives and Response Plan**-- Are NIH's responses to
    risk aligned with our tolerance for risk?

-   **Control Activities, Assurance** **and Testing**-- Are risk
    controls clearly defined in policies and procedures?

-   **Risk Intelligence, Communication** **and Training**-- Is key risk
    intelligence gathered, interpreted and communicated?

-   **Monitoring and Escalation** -- Are enterprise risks periodically
    monitored and is a proper escalation process in place?

-   **Sustainability and Continuous Improvement** -- Is risk
    intelligence sustained in an integrated and coordinated manner? Are
    risk management processes continuously reevaluated and improvements
    implemented?

+-----------------------------------------------------------------------+
| **Key Observations from the Study**                                   |
+=======================================================================+
| -   NIH has an executive office structure composed of several offices |
|     of the director that manage extramural and intramural research,   |
|     technology, special and emerging issues, administrative and       |
|     financial programs. The Office of the Director is supported by    |
|     various advisory groups. The administrative, financial and        |
|     support offices, as well as certain emerging issues offices and   |
|     centers report to the Deputy Director, who in turn reports to the |
|     NIH Director. Directors lead over 27 institutes and centers and   |
|     report to the NIH Director. Many institutes and centers have      |
|     their own extramural, intramural, financial, performance,         |
|     technology and other administrative offices, divisions and        |
|     branches to support specific institute and center mission         |
|     requirements.                                                     |
+-----------------------------------------------------------------------+
| -   NIH employs four types of Advisory Committees:                    |
|                                                                       |
|     -   Integrated/Initial Review Groups (IRGs) and Special Emphasis  |
|         Panels (SEPs)                                                 |
|                                                                       |
|     -   Boards of Scientific Counselors (BSCs)                        |
|                                                                       |
|     -   Program Advisory Committees (PACs)                            |
|                                                                       |
|     -   National Advisory Councils and Boards (NACs)                  |
+-----------------------------------------------------------------------+
| -   Committee management at NIH resides at the NIH Office of the      |
|     Director (OD) level in the Office of Federal Advisory Committee   |
|     Policy and the Institute or Center (IC) level with the Committee  |
|     Management Office (CMO). OFACP is responsible for the oversight   |
|     of all NIH Federal advisory committees under the auspices of the  |
|     FACA.                                                             |
+-----------------------------------------------------------------------+
| -   At the IC level, each IC has a CMO or uses the resources of a     |
|     service center, to support the committee management function      |
|     within the Institute or Center.                                   |
+-----------------------------------------------------------------------+
| -   A program management office (PMO) has been established to set the |
|     direction of primary ERM activities and to promote a coordinated  |
|     and leveraged approach to ERM across NIH.                         |
+-----------------------------------------------------------------------+
| -   Senior NIH Management is beginning to be effectively engaged in   |
|     the ERM process, serving in some aspects of a risk oversight      |
|     council.                                                          |
+-----------------------------------------------------------------------+
| -   Lists of risks may or may not reflect interdependences or be      |
|     mapped to processes and systems, be driven by data where          |
|     feasible, or be linked to standard organization and               |
|     process/system representations of NIH.                            |
|                                                                       |
| -   Some controls appear to be appropriately tested across key        |
|     processes, systems and functions throughout NIH.                  |
+-----------------------------------------------------------------------+
| -   Deficient conditions appear to be identified as a result of       |
|     assurance activities. Conditions appear to be promptly            |
|     investigated and corrective actions seem to be formulated.        |
+-----------------------------------------------------------------------+
| -   NIH currently uses management control based areas of risk.        |
+-----------------------------------------------------------------------+
| -   In some offices, current lists of risks are generated by          |
|     reference to typical events, recent experiences within and        |
|     outside of NIH.                                                   |
+-----------------------------------------------------------------------+
| -   Various NIH offices and other organizational units employ         |
|     checklists to self-assess and self-report risks.                  |
+-----------------------------------------------------------------------+
| -   Some units use severity and probability as key dimensions for     |
|     risk assessment                                                   |
+-----------------------------------------------------------------------+
| -   NIH has various detailed programs of improvement resulting from   |
|     activities that may include for example: internal reviews,        |
|     internal or external assessments, user feedback, complaints and   |
|     other issues.                                                     |
+-----------------------------------------------------------------------+
| -   NIH does not appear to have an NIH-wide approach for aggregating, |
|     reporting, or responding to risks or generating risk response     |
|     alternatives.                                                     |
+-----------------------------------------------------------------------+
| -   Many of offices report ad hoc approaches to generating risk       |
|     responses, and some highly specialized approaches to mitigating   |
|     risks (e.g., ORF, Acquisitions, Technology).                      |
+-----------------------------------------------------------------------+
| -   NIH is in the midst of changing its risk approach and harmonizing |
|     the generation of risk response To focus more on                  |
|     preventing/mitigating risk.                                       |
+-----------------------------------------------------------------------+

## Summary of Recommendations

+-----------------------------------------------------------------------+
| **Key Recommendations of the Study**                                  |
+=======================================================================+
| **Governance and Risk Oversight Recommendations**                     |
|                                                                       |
| 1.  The Advisory Committee should annually review and approve NIH     |
|     risk management policy. This policy should clearly define risk,   |
|     NIH's philosophy about risk, its risk tolerance, its appetite for |
|     various types of risks and the relationship between mission       |
|     critical risks and stakeholder value drivers.                     |
|                                                                       |
| 2.  The Advisory Committee should approve a risk management policy    |
|     that outlines the mechanisms to delegate authorities and to       |
|     elevate issues and conflicts.                                     |
|                                                                       |
| 3.  The Advisory Committee should review NIH's budgetary and other    |
|     financial objectives to ensure compatibility with the level of    |
|     risk embedded in NIH's annual plan.                               |
+-----------------------------------------------------------------------+
| **Risk Identification** **Recommendations**                           |
|                                                                       |
| 1.  Executive management should identify the key performance drivers  |
|     of NIH and the associated risks and scenarios by process, system, |
|     budget area, organization, mission objective, and other relevant  |
|     segments such as applicable laws, regulations and policies.       |
|                                                                       |
| 2.  Taxonomy of risks relevant to NIH's activities, industry          |
|     participation, and mission objectives should be updated regularly |
|     based on the volatility of NIH's objectives, budget, personnel    |
|     and its operating, regulatory and stakeholder environment.        |
|                                                                       |
| 3.  The risk taxonomy should be revised based on emerging and         |
|     experienced risks on at least an annual basis, ratified by the    |
|     NIH Director and the NIH Advisory Committee.                      |
|                                                                       |
| 4.  NIH Management should lead an effort to integrate risk management |
|     into the NIH culture                                              |
+-----------------------------------------------------------------------+
| **Risk Assessment and Measurement**                                   |
|                                                                       |
| 1.  NIH-wide risks should be assessed on the basis of gross impact    |
|     (ie., gross risk, inherent risk, before mitigation and controls), |
|     net impact (ie, net risk, residual risk, vulnerability- after     |
|     mitigation and controls).                                         |
|                                                                       |
| 2.  Assessments can be a combination of qualitative and quantitative  |
|     and should be facilitated externally (for objectivity &           |
|     independence) not simply self-administered and self-reported.     |
|                                                                       |
| 3.  Wherever possible NIH should drive assessments with process,      |
|     system, budgetary, and external factor data, as available from    |
|     independent and controlled sources.                               |
|                                                                       |
| 4.  Assessment scales should be founded on performance and risk       |
|     indicators, and where not available, based on consistently        |
|     constructed data-equivalent word descriptions of events.          |
|                                                                       |
| 5.  Gross risk (before risk response and controls) should be assessed |
|     at least in relation to quantitative factors such as process      |
|     contribution, organization size, workflow volume (e.g., grants,   |
|     facilities) and cost of prior risk experience (direct hits and    |
|     near misses), as well as qualitative factors such as speed of     |
|     onset of the threat, as well as impacts on key stakeholders,      |
|     reputation, legal/regulatory, environment, health and safety and  |
|     speed of onset.                                                   |
|                                                                       |
| 6.  Net risk or vulnerability (after controls) should be assessed in  |
|     relation to such factors as Control Effectiveness (especially     |
|     relative to People, Process and Systems) Speed of Response        |
|     (Detection, Response, Recovery), Complexity or volatility of      |
|     activities, and Geographical dispersion, Response to Prior Risk   |
|     Experience, Rate of Internal Change and External Conditions.      |
|                                                                       |
| 7.  Net risk should be assessed relative to risk appetite and further |
|     mitigated as needed                                               |
|                                                                       |
| 8.  Mitigated value should be calculated, i.e., the difference        |
|     between gross and net risk.                                       |
|                                                                       |
| 9.  Internal and external subject matter specialists should be        |
|     involved as appropriate in the assessment of risk.                |
|                                                                       |
| 10. NIH's exposure to extreme upsides and downsides should be         |
|     assessed on a regular basis relative to intolerable and tolerable |
|     risk categories, for tolerable risks according to NIH risk        |
|     appetite.                                                         |
|                                                                       |
| 11. Probabilistic analyses should only be used when appropriate,      |
|     e.g., when there is operative a law of large numbers, and cause   |
|     and effect relationships are known, as well as other assumptions  |
|     required for meaningful use of probabilistic analysis.            |
|                                                                       |
| 12. Statistical risk measures should always be supplemented with      |
|     stress-testing and scenario analysis especially for extreme gross |
|     and net risks.                                                    |
|                                                                       |
| 13. Risk measures should be adapted to the types of risks taken       |
|     especially in relation to the mission specific objectives of      |
|     offices, centers and institutes and their use of otherwise common |
|     processes and systems.                                            |
|                                                                       |
| 14. Root Cause Analysis should be performed when significant risks do |
|     occur.                                                            |
|                                                                       |
| 15. Estimates of gross and net risk should be revised based on        |
|     experience                                                        |
|                                                                       |
| 16. A systematic independent verification of the risk assessment      |
|     process and results should be used to validate relevance and      |
|     efficiency.                                                       |
+-----------------------------------------------------------------------+
| **Response Alternatives and Response Plan Recommendations**           |
|                                                                       |
| 1.  Alternative risk responses should be formulated and evaluated for |
|     alignment with the NIH\'s risk tolerance and risk appetite.       |
|                                                                       |
| 2.  Risk responses should be organized in a hierarchy of response,    |
|     i.e., from lowest to highest strength of response, cost, and      |
|     other factors consistent with NIH policy, risk appetite and       |
|     culture.                                                          |
|                                                                       |
| 3.  NIH management should determine priorities taking into account    |
|     such factors as speed of risk onset, urgency, cost of mitigation  |
|     compared to expected benefit, degree of difficulty and time       |
|     required to implement.                                            |
|                                                                       |
| 4.  Responses should be integrated to provide effective and timely    |
|     NIH-wide preparation, response and recovery.                      |
+-----------------------------------------------------------------------+
| **Control Activities, Assurance** **and Testing Recommendations**     |
|                                                                       |
| 1.  Management should identify appropriate control activities to      |
|     ensure that its risk responses are carried out properly and in a  |
|     timely manner for all mission-critical risks at NIH.              |
|                                                                       |
| 2.  Mandatory disclosure requirements should be met in a timely       |
|     fashion.                                                          |
|                                                                       |
| 3.  General and application controls, including preventive,           |
|     detective, manual, computer and management controls should be     |
|     clearly defined in policies and procedures for each               |
|     mission-critical risk, relative to each process/system segment,   |
|     organizational unit, and budgetary authority.                     |
|                                                                       |
| 4.  Control activities should be matched to the speed of risk onset,  |
|     not the speed of response.                                        |
+-----------------------------------------------------------------------+
| **Risk Intelligence, Communication** **and Training Recommendations** |
|                                                                       |
| 1.  Risk intelligence should be gathered internally and externally.   |
|                                                                       |
| 2.  Risk intelligence should be evaluated for systemic bias, e.g.,    |
|     credibility of the source (including trustworthiness and          |
|     competence).                                                      |
|                                                                       |
| 3.  Risk intelligence should be available on a timely basis relative  |
|     to speed of risk onset.                                           |
|                                                                       |
| 4.  Risk intelligence should be integrated within and across core     |
|     decision-making processes and risk specializations.               |
|                                                                       |
| 5.  Performance Metrics / Dashboards / Scorecards should be           |
|     established for all mission-critical risks, especially as they    |
|     are segmented by processes and systems, relative to               |
|     accountabilities and incentives for improvement. Metrics may be   |
|     both qualitative and quantitative. Each scorecard element should  |
|     include NIH's risk appetite, actual current results, previous     |
|     results and target metric.                                        |
+-----------------------------------------------------------------------+
| **Monitoring and Escalation** **Recommendations**                     |
|                                                                       |
| 1.  NIH management, risk personnel and risk oversight groups should   |
|     monitor mission critical risks to review and evaluate performance |
|     compared to NIH objectives.                                       |
|                                                                       |
| 2.  NIH management should monitor mission-critical risks to review    |
|     and evaluate adherence to applicable laws, regulations, policies  |
|     and procedures.                                                   |
|                                                                       |
| 3.  NIH management should monitor mission-critical risks to review    |
|     and evaluate whether or not substantial progress is being made in |
|     managing risk exposures so that they are within NIH's appetite    |
|     for such exposures, if tolerable.                                 |
+-----------------------------------------------------------------------+
| **Sustainability and Continuous Improvement** **Recommendations**     |
|                                                                       |
| 1.  Lessons learned should be identified and communicated to          |
|     appropriate personnel on a timely basis.                          |
|                                                                       |
| 2.  Failures to correctly identify and assess risks should be         |
|     investigated and immediately remediate.                           |
|                                                                       |
| 3.  Risk related policies and procedures should be reviewed and       |
|     updated on a timely basis.                                        |
+-----------------------------------------------------------------------+

## Summary of Answers to Study Questions

During this study, the following questions are answered:

> *1. What is the best methodology for the NIH to use to evaluate risk
> for its scientific, administrative, and financial programs for the
> NIH-wide risk assessment in Phase 2, and in the future?*

-   Leading practice methodology for NIH exists in form of maturing
    standards and guidance, and examples of implementations at large,
    complex organizations

-   The methodology includes governance, risk identification, risk
    assessment and measurement, risk response alternatives and plans,
    monitoring and escalation, control activities, assurance and
    testing, risk intelligence, communications and training, and
    sustainability and continuous improvement

-   These methodology components are listed using the 8 dimensions of
    the risk intelligent organization, are verified though a preliminary
    survey of over 120 organizations and case example

> *2. What is the most economical, efficient and effective way to
> collect the data needed to evaluate these high risk areas without
> imposing an excessive burden on program staff or NIH administrators?*

-   Based on leading practice methodology components a 5 step process is
    proposed

-   The process is a facilitated assessment and can be implemented using
    3 options, also presented, in order of increasing data, information
    , and intelligence-driven sophistication

-   Using the 3 options the 5 step process can be economically,
    efficiently, and effectively deployed in a scalable fashion across
    all of NIH and within individual institutes, centers and offices

-   Only by embedding this process into the regularly occurring NIH
    strategy, operations, scientific, financial, budgeting, and
    administrative process and system

> *3. What is the baseline status of the existing risk areas previously
> identified by NIH managers?*

-   This study, through media scans, reference to subject matter
    specialists, review of NIH existing documentation of risk areas and
    other documents, and interviews with NIH executives has produced a
    list of baseline risk examples for review by NIH

-   Among other areas, processes and systems associated with grants,
    human resources, emergency preparedness, strategic planning and
    budgeting appear to be mentioned by NIH executives as very high
    priority risk areas for consideration

# 1. Introduction

## 1.1 Overview of NIH-wide Risk Assessment

*It is important to note that this assessment was performed in
accordance with the Standards for Consulting Services established by the
American Institute of Certified Public Accountants. Classified as such,
the purpose of this report is to provide observations, conclusions and
recommendations for improvement of business execution to NIH Senior
Management for their consideration. Moreover, this assessment does not
constitute an audit made in accordance with U.S. generally accepted
accounting or auditing standards (GAAP or GAAS), the objective of which
is the expression of an opinion on the elements, accounts, or items of a
financial statement. Therefore, Deloitte & Touche LLP is not in a
position to express, and will not express, an opinion, or any other form
of assurance, with respect to any matters as a result of performing this
assessment. Moreover, adherence to industry prevalent or leading
practices as described in this document does not provide any level of
assurance that control breakdowns have not or will not occur that could
result in materially significant losses or weaknesses and deficiencies
to NIH.*

*Accordingly, our work specifically did not include:*

-   *An evaluation of the [appropriateness]{.underline} and
    effectiveness of NIH strategies, operations, transactions or data;*

-   *Benchmarking of risk and return performance;*

-   *Performing [detailed tests of compliance]{.underline} or
    [transaction testing]{.underline} to determine that controls are
    operating in accordance with their design;*

-   *Performing an [independent valuation]{.underline} of transactions
    or validating quantitative methods or calculations;*

-   *Performing [tests of system functionality]{.underline} or of
    general system and application level controls;*

-   *A [specific evaluation]{.underline} of human resource skills
    especially in performing risk assessment or management functions;*

-   *[Implementation analysis]{.underline} of any comments or
    recommendations.*

*This Study provides an overview of the key findings and recommendations
of methodology assessments.*

### 1.2 Purpose

As the nation\'s medical research agency, the NIH makes important
medical discoveries that improve health and save lives. For an
organization such as the NIH, effective risk management extends beyond
traditional risk function to critical areas that preserve and promote
the NIH's mission. The NIH's unique mission of "science in pursuit of
fundamental knowledge, and the application of that knowledge to extend
healthy life and reduce the burdens of illness and disability" demands a
more comprehensive and active view of risk management, beyond compliance
and controls. Consistent with Committee of Sponsoring Organizations
(COSO), risk is not only as a loss event, but a potential loss from
failing to meet mission objectives. Given NIH's unique mission, an
enhanced risk management program will allow the NIH to improve program
effectiveness and prevent mission failures, thereby allowing the NIH to
preserves its resources on the achievement of the NIH's mission.

The goals of the NIH's enhanced Risk Management Program include the
expansion of the NIH's current processes for meeting FMFIA and OMB
Circular A-123 requirements to perform periodic risk assessments. The
NIH desires to enhance its existing Risk Management Program in order to
eventually evaluate risk across its organization, including
administrative, operational, scientific and financial programs.

These goals are consistent with basic elements of Enterprise Risk
Management (ERM) -- an integrated set of tools for improving "risk
intelligence" and enhancing the achievement of program objectives. Risk
Intelligence encompasses both "unrewarded risk" (asset protection) as
well as "rewarded risk" (value creation) in NIH's case, "value creation"
translates in to the ability & capability to better leverage resources,
navigate an uncertain environment in order to fulfill Institute
strategic objectives.

### 1.3 ERM Applied to NIH

ERM, broadly speaking, has been in existence at least a decade. In some
industry sectors, notably financial services and energy, most
industry-specific risks are managed with a high level of finesse, using
complex probabilistic modeling and sophisticated analyses. Other
organizations, especially those in services and consumer business
sectors, may have a less refined approach to risk management, and the
need for more systematic practices is just now emerging.

But it is the rare organization that completely manages the full
spectrum of risk, that:

-   Adequately assesses and addresses risk from all perspectives and
    programs;

-   Breaks through the organizational barriers that obscure a view of
    the entirety of risks facing an organization; and

-   Systematically anticipates and prepares an integrated response to
    potentially significant risks.

When properly implemented and adopted, ERM enables an organization to
improve its risk intelligence and enhance program performance.

ERM, in its broadest sense, will enable the NIH to enhance its Risk
Intelligence by achieving:

-   Risk management practices that will benefit the entire NIH
    organization, creating connections between administrative,
    operational, financial, scientific, and programs areas so that risk
    can be viewed holistically across the NIH, not just in the so-called
    "silos" that often arise within large, mature, and/or diverse
    organizations

-   Risk management strategies that address the full spectrum of risks -
    strategic, execution and operational. These include mission-specific
    strategies associated with various process and systems including
    those that deal with the safety of human and animal subjects,
    financial (the appropriate use of funding), compliance,
    environmental, security, privacy, business continuity, strategic,
    reporting (transparency and validity of experimental data,
    operational, and other areas

-   Risk assessment processes that augment the conventional emphasis on
    probability by placing significant weight on vulnerability which can
    better prepare the NIH for improbable, but potentially catastrophic
    impacts

-   Risk management approaches that do not solely consider single
    events, but also take into account high risk scenarios and the
    interaction of multiple risks across multiple processes an systems
    which is more appropriate for a complex organization such as the
    NIH.

-   Risk management practices that are infused into the organization's
    culture, with a common risk taxonomy and risk appetite well defined,
    so that strategy and decision-making evolve out of a risk-informed
    process, instead of having risk considerations imposed after the
    fact (if at all)

-   Risk management philosophy that focuses not solely on risk
    avoidance, but also on risk-taking as a means to create value for
    the organization's various stakeholders.

### 1.4 Background

To bring the NIH the most comprehensive approach based on leading
practices, the risk assessment team leverages extensive A-123
experience, as well as a customized approach to Risk Intelligence in the
form of governance & risk oversight, enterprise risk assessment, risk
identification and mitigation, monitoring and escalation, control
assurance, and sustainable continuous improvement.

The approach is the result of over a thousand SOX 404, A-123, SAS70 and
internal control assessments; the implementation of over a thousand SAP
and Oracle ERP software implementations, comprehensive technology change
management and human capital solutions, including many shared services
providers that have organizational structures and responsibilities
similar to those of NIH.

While the NIH has chosen to address these elements, these are joined
with the other building blocks of a comprehensive methodology, the
*Eight Dimensions of Risk Intelligence and Capability,* which include:

-   Governance, Risk Identification

-   Risk Assessment and Measurement

-   Response Alternatives & Response Plan

-   Control Activities

-   Assurance and Testing

-   Risk Intelligence

-   Communication and Training

-   Monitoring and Escalation

-   Sustainability and Continuous Improvement.[^6]

Within these 8 dimensions, there are 89 sub-categories based on global
leading practices that incorporate and integrate for example, COSO I and
II (ERM), Australia New Zealand Standard 4306:2004, Kontra G, Turnbull,
ISO14000, King, Dey, Standard & Poor and Moody's. Each sub-category
contains entity level considerations to enable assessment of current
state versus desired state, tests of capability, and appropriateness of
design, effectiveness, efficiency and responsibility.

The NIH has stated a need to improve the effectiveness and efficiency of
its risk management program. The NIH is concerned about reducing the
burden of risk management on the business of the NIH. The methodology
identifies opportunities for rationalization, synchronization and
harmonization of key risk management themes (including risk
identification & response).

To deploy an enhanced enterprise risk management service NIH will
require the following:

-   Ability to directly link risks to mission and program objectives

-   Establish a highly experienced team across a broad range of sectors
    such as healthcare, life sciences, not-for-profit and federal
    government

-   Ability to rapidly adapt methodologies and tools to the specific,
    yet variable requirements given the need to integrate with both a
    scientific/research based culture and its administrative and
    financial operations

-   Ability to minimally intrude on and burden the business based on a
    highly efficient methodology that is technology enabled with a
    streamlined data gathering process.

-   Ability to initiate transfer of knowledge to NIH for streamlined and
    sustainable capability, especially scalable to the many diverse NIH
    offices, centers and institutes

-   Ability to establish and maintain relationships with senior
    executives in a variety of scientific, financial, and administrative
    contexts

-   Communication and facilitation skills that supports active
    engagement of NIH stakeholders in the risk intelligence process

-   Ability to provide end-to-end, comprehensive solutions for
    identified risks

-   Deep industry expertise in non-for-profit, health care, life
    sciences and performance measurement/data collection

In addition to these overall factors, NIH requires that the NIH-wide
risk assessment methodology should:

-   Be applicable to the diversity of processes, systems and
    organizational objectives comprising NIH

-   Focus on the mission critical risks to NIH performance against
    mission and especially worst case, weakest link, and worst nightmare
    scenarios

-   Focus on process and system risk

-   Be data-driven to the extent possible

-   Be able to identify and assess risks for disposition within the
    initial steps of the methodology so immediate corrective actions can
    be taken as early as possible

#   

# 2. Study Design and Methodology

## 2.1 Conceptual Framework of the Study

The goal of this feasibility study ("study") is to provide a
scientifically sound[^7] design for an enterprise-wide risk assessment
("assessment") methodology.[^8] By "scientifically sound" is meant a
methodology that is grounded in scientific knowledge, that is, both
reason and experience. By "methodology" is meant a documented approach
for performing risk assessment tasks in a consistent within NIH, U.S.
government and leading practice norms, relevant to NIH, comprehensive in
view of NIH-wide risks, and coherent to the culture and traditions of
NIH. "Sound scientific design" for a methodology is a set of risk
assessment rules, process, procedures that is grounded in scientific
knowledge. Scientific knowledge embeds logical validity and empirical
verifiability in the risk assessment methodology.

In this study, *logical validity* is achieved by a rigorous examination
of guidance and standards of risk assessment across several
jurisdictions and domains of practice, and application of risk
assessment principles to the design of the NIH-wide risk assessment
methodology. A survey of recent research from various social and
decision sciences and a review of accumulated guidance from regulators
can provide a preliminary foundation for the logic of the methodology.
*Empirical validity* can be achieved by reviewing the experience of
organizations that have recently implemented enterprise-wide risk
assessments. This experience can be collected through web-based survey
of practices and by case study examples of recent implementations.[^9]

The study was prompted by National Institutes of Health ("NIH")
Director's interest in establishing sound management leading practices
in risk management across NIH. This interest is supported by the GAO
High Risks report of an emerging area for all Federal agencies, "that
instilling a disciplined approach to identifying and managing risk has
broad applicability across a wide range of federal programs, operations,
and functions across the federal government. This will be a continuing
focus of our work in the future.[^10]

This study provisionally answers three questions:

1.  What is the best methodology for the NIH to use to evaluate risk for
    its scientific, administrative, and financial programs for the
    NIH-wide risk assessment in Phase 2, and in the future?

2.  What is the most economical, efficient and effective way to collect
    the data needed to evaluate these high risk areas without imposing
    an excessive burden on program staff or NIH administrators?

3.  What is the baseline status of the existing risk areas previously
    identified by NIH managers?

The short answer to question one is that several methodologies exist and
have been used by various organizations over the past several years.
This study will systematically assess the methodologies that have been
used to ascertain a set of practices useful for NIH to consider in
assessing and managing enterprise risk. The study will also consider
approaches that offer fewer burdens on the business of operating NIH
programs. In the course of the study several risks can be identified to
suggest a baseline for commencing an assessment of NIH risks to the
performance of NIH mission.

This study framework and description of risk management practices is
based on Deloitte & Touche LLP's extensive knowledge base of governance
and transaction processing control activities employed at major entities
that operate within - and manage risks in relation to various
industries. These entities include active and established participants
in the global energy marketplace, as well as large financial
institutions, manufacturers, consumer businesses, highly regulated,
public and private sector organizations. There are limitations to the
meaningfulness of comparisons and extractions of leading practices due
to the relatively underdeveloped risk management and control
capabilities of many organizations and the continually evolving nature
of various industries, regulation, and risk management. Given that
meaningful gaps between current practices and desired capabilities
exist, implementing practices can leave an enterprise in good company,
but not necessarily in good comfort. This study attempts to provide
contextual discussion highlighting this dynamic.

The study does not assume that there is a unique "best" risk management
practice available for any organization, particularly NIH. However,
there are several "leading practices" for NIH consideration. A "leading
practice" represents practices from across private and public sector
organizations, including regulators that may be considered to provide a
strong level of capability to identify, assess, respond to, monitor,
control risks as they can occur in complex organizational processes and
systems. *Leading practices*, by definition, are aspirational, and need
to be viewed not just within the context of cost versus benefits
provided, but relative to the culture of NIH, its current constraints
and mission objectives.

For the purposes of this study, risk is a possible loss due to failure
of NIH to perform against its mission. Failure can emanate from any
combination of factors, including those generated by people, process,
systems, technology, science, or occasioned by external events. The goal
of an enterprise risk assessment is to prioritize risks, segmented by
their location of occurrence (e.g., process and system, organizational
unit), in order to effectively and efficiently allocate resources to
respond to specific risks.

Several constraints make scientific study of risk management methods
challenging[^11]:

-   The real values for loss are not known, or may even be unknowable *a
    priori*.

-   Each set of events occurring to an organization is unique and not
    repeatable; worse: ensembles of events are unpredictable.

-   Risk management method cannot be separated from the object of study:
    if a method results in some action, the state of the organization
    that employs the method irrevocably changes.

-   Introduction of a risk management method to an organization changes
    the behavior of participants in an organization, as well as the way
    participants may deploy processes and systems in response to
    systematic knowledge about potential losses.

-   A single occurrence of a risk, whether predicted or not, cannot be
    used to draw any conclusions about the accuracy of risk analysis
    methods.

-   The objects of an NIH-wide risk assessment (strategies, processes,
    budgets, resources) have relatively long cycle times.

The above constraints may limit the empirical study design options
available in risk management to answer the three questions.[^12] These
constraints do not prevent an application of systematic, logical and
empirical scientific principles. Recognition of these constraints allows
a design to provide more reliable results than anecdotal descriptions of
risk assessment practices.

The figure below aids the discussion of appropriate choice of techniques
to accomplish the goals of the study. Two primary dimensions describe
the type and frequency of event data the study can employ. Based on the
list of study constraints, there appears to be a relative low frequency
of useful data available, and the data are occurring in a currently
unstable, potentially unrepeatable environment. Thus the leading choices
include context-based techniques including expert opinion and case
studies. These should be supplemented with logical models and
sensitivity analysis.

**Figure 1. Study Method Choices**

![](media/image1.wmf){width="4.71875in" height="3.3055555555555554in"}

Given that study techniques are constrained by the type and frequency of
data for the study, the organizing principle used to address the study
questions is a conceptual framework that incorporates multiple
approaches to data and analysis.[^13] These approaches are used to
converge on assessment methodology practices based on the principle of
relating similar practices with one another. Thus the approach
"triangulates" the analysis to provide provisional conclusions regarding
leading enterprise risk assessment practices:

1.  Review of risk-informed decision making literature

2.  Existing risk assessment guidance

3.  Exploratory survey of risk assessment practices

4.  Case examples of recent enterprise risk assessment developments

5.  Review of NIH documents, data and interviews of NIH executives

An outline of the feasibility study design is depicted in Figure 2.
Study questions, assumptions and constraints feed into baseline data
gathering from NIH documents, risk literature, and risk standards and
guidance. With this data the eight dimensions of a risk intelligent[^14]
organization are developed and used to inform NIH interviews, construct
case studies and interpret preliminary Enterprise Risk Management
("ERM") survey results. Interviews, document, literature and standards
review, case studies, and preliminary survey results then provide the
basis for answering the three study questions.

**Figure 2. Feasibility Study Design**

![](media/image2.wmf){width="6.0in" height="2.8645833333333335in"}

Through multiple data sources, the study is less vulnerable to
overlooking risk assessment practices, their implementation and
potential for transferability to NIH.

## 2.2 Data Collection and Analysis Procedures

Data collection starts with a survey of recent research from the fields
of social psychology, decision sciences, economics, and political
science. Extensive surveys already exist which makes search for sources
more efficient. The purpose of this research survey is to incorporate
into a methodology design for NIH, where possible, relevant components
of group and facilitated decision making insights from these
disciplines. From this survey a catalogue that summarizes these insights
is compiled. The catalogue of insights is then incorporated into a
leading practice design for a risk assessment methodology. These
insights often represent risk intelligent decision process pitfalls to
be avoided and lessons learned from social science decision experiments
of decision making under uncertainty.

The analysis of ERM methodology components continues by obtaining and
analyzing documentary sources of practices. These sources include
standards, guidance, and examples of existing methodologies, surveys of
organizations, and case studies of recent implementations.

There are several existing sets of guidance available to the designer of
a risk assessment methodology. They include:

-   Committee of Sponsoring Organizations ("COSO", formerly the Treadway
    Commission)

-   Standards Australia and Standards New Zealand

-   International Organization for Standardization ("ISO")

-   Basel II Capital Accords

-   H.M Treasury

-   Federal Reserve System

-   Office of the Controller of the Currency

-   Presidential/Congressional Commission on Risk Management

-   National Institute of Standards and Technology

Other sources along with these are listed in the Appendix. Along with
regulatory guidance there are extant practices and guidance, often
sponsored by the public sector, from diverse areas such as

-   Institute of Internal Auditors

-   New York Stock Exchange

-   OCTAVE® (Software Engineering Institute, Carnegie Mellon University)

The development of a set of practice categories to represent risk
management methodology begins with an initial categorization of
practices using the COSO ERM guidance. The labels for these categories
were then modified to represent the Standards Australia/Standards New
Zealand. With these categories as a starting point, a list of practices
by category is compiled. After lists of practices by category have been
compiled, a set of category key words and an operational definition is
developed. Key words are unique sets of nouns, verbs, adjectives and
adverbs that are unique to the category. Using the category key word and
operational definition, a check for appropriate category is conducted
using the following guidelines:

-   Each practice must not overlap with another practice

-   Each practice is assigned to only one category

-   Each practice should have words related to or include key words
    associated with the category

Finally, a review committee of practicing auditors and risk management
practitioners review the categories, assignment of practices,
definitions and key words for consistency, completeness, alignment with
practice and application of professional judgment.

The result is a set of :

-   8 risk intelligence dimensions: governance, risk identification,
    risk assessment and measurement, risk response alternatives and
    plans, monitoring and escalation, control activities, assurance and
    testing, risk intelligence, communications and training,
    sustainability and continuous improvement;

-   89 sub-dimensions distributed across the 8 dimensions of risk
    intelligence;

-   Over 512 considerations distributed over the 89 sub-dimensions of
    risk intelligence.

-   Representing guidance and standards from over 15 separate sources,
    including COSO II (ERM), ANZ 4360:2004, and others.

To supplement the practices gleaned from several sets of risk management
standards and guidance, an exploratory survey of enterprise risk
management practices at organizations was conducted. This is a web-based
survey open to organizations by invitation of the study authors. The
survey was conducted over the past 8 months and represents the perceived
state of enterprise risk management in over 120 organizations, publicly
and not publicly traded, in the U.S. and Europe. The results are highly
preliminary, but indicative of the guidance and standards available at
this time. The study uses results of the distribution of responses to
develop implications for enterprise risk management methodology.

The choice of case examples[^15] of ERM practice is affected by the time
at which the ERM program was initiated and the case information
recorded. ERM has both a pre- and post-Sarbanes-Oxley Act manifestation.
Sarbanes-Oxley was a major intervention in the ERM marketplace and had a
significant impact on corporate governance, executive compensation, the
state of internal controls, compliance with regulation, and many other
areas of risk to value for publicly traded companies. The deployment of
OMB Circular A-123 has had similar ramifications in degree for U.S.
government entities.

Prior to Sarbanes-Oxley, ERM was treated by many companies as an
enhancement to decision process, corporate governance, and prudent
management practice. After Sarbanes-Oxley year one audits, ERM has
gained a markedly different character. It may be viewed as a way to
better govern and manage mission critical risk, one instance of which is
material mis-statement, another of which is integrity breaches by
organizational leadership. For those companies listed on the New York
Stock Exchange, further requirements delimited Audit committee
responsibilities regarding risk management especially regarding
financial disclosures and exposure.[^16] Accordingly, examples of
enterprise risk assessment methodologies are gathered for
implementations post-Sarbanes-Oxley.

The criteria for a case to be included are informed by the character of
NIH as a regulated, large, complex organization:

-   Formality of assessment program (required by organizational policy)

-   Organizational size (\> \$5 billion in revenue and or \> 5,000
    employees)

-   Complexity of operation (more than two business units, each unit
    contributing at least 20% of revenue or budget)

-   Government stakeholder (at least 10% of revenue deriving from
    government grants or contracts)

Three recent cases of enterprise risk assessment implementation are
developed using the 8 dimensions of risk intelligence to profile each
case.

### 2.3 NIH Documents

Office of Management Assessment has made available the following
documents:

-   Recent management control and risk area assessments at NIH

-   Recent GAO reports

-   Recent third party reports on NIH assessment, operations, and
    systems

-   Organizational charts

-   Other organizational data

In addition, the study relies on the documents listed at several sites
including:

-   [www.nih.org](http://www.nih.org/)

-   [www.gao.org](http://www.gao.org/)

-   [www.whitehouse.gov](http://www.whitehouse.gov/)

### 2.4 Open Source Searches

The study relies on risk management practice data contained in various
online repositories maintained by governmental and non-governmental
organizations. Examples include:

-   [www.riskworld.com](http://www.riskworld.com/)

-   [www.riskportal.com](http://www.riskportal.com/)

-   [www.cabinet.gov.uk](http://www.cabinet.gov.uk/)

-   [www.bis.org](http://www.bis.org/)

-   [www.coso.org](http://www.coso.org/)

-   [www.federalreserve.gov](http://www.federalreserve.gov/)

-   [www.nist.gov](http://www.nist.gov/)

-   [www.cmu.sei.edu](http://www.cmu.sei.edu/)

### 2.5 NIH Site Visits

The data in this study was enhanced by site visits with NIH directors
from:

-   Office of the Director

-   Office of Management

-   Office of Management Assessment

-   Office of Intramural Research

-   Office of Extramural Research

-   Various other offices and centers at NIH, a complete listing is
    provided in the Appendix

In addition to NIH site visits, the study relies on interviews with
members of organizations represented by the case studies. These
interviews are confidential sources and the identity of these sources
may not be disclosed.

# 3. Assessment of ERM Methodologies

### 3.1 Overview

The demands on organizations such as NIH to develop risk intelligence
capability are increasing. Organizations are becoming more complex, the
risks to realizing the stakeholder value are amplified, speed of onset
of risks is shortened, and the need to develop harmonized risk-related
processes and systems at reasonable cost is paramount. In this context,
effective and efficient process and systems are critical factors for
risk management development. To deal with these problems, it becomes
essential to provide tools to support organizations to perform their
tasks. To be effective, however, these tools must work together.

Harmonization of risk-related processes and systems has been considered
one of the most challenging issues for the risk governance environment.
Harmonization demands consistent representations of organizational and
risk information, standardized processes that link organizational units,
stakeholders, processes and systems, homogeneous means of communication
among stakeholders, organizations and risk owners. In this context, it
becomes necessary to build an infrastructure for risk assessment and
management harmonization. This infrastructure should be based on robust
conceptual models deployed across organizations over time and culture.
Risk management tools spread out over complex organizational units must
process a common set of risk data and thus must share a common
understanding of what the risk data mean.

Harmonization of risk process and product becomes a particularly
important for an organization such as NIH for many of the following
reasons:

-   NIH has a leading edge science of health mission, not easily
    comparable to other institutions and private sector organizations;

-   NIH has experienced doubling of its budget and activities to support
    its mission in the past 5 years;

-   NIH is composed of an Office of Director (with over 2,000 staff) and
    27 institutes and centers (with staff of over 17,000), all leaders
    in their respective fields, each with mission specific risks,
    opportunities, and requirements;

-   Over 80 percent of NIH's budget is allocated as funds granted to
    external research entities such as private laboratories and medical
    centers, academic institutions, and other government agencies;

-   The NIH Bethesda campus with over 300 acres (along with the adjacent
    Naval Medical Center and soon the addition of Walter Reed Hospital)
    comprise one of the most densely populated sites among all Federal
    facilities;

-   Mission critical positions comprise over 30% of the staff functions,
    and over 2/3rds of these functions are staffed by individuals
    eligible for retirement in 3 to 5 years

In this context, [risk intelligence]{.underline} for NIH is the process
and the capability of gathering, understanding, monitoring, reporting,
and responding to risks to performance of NIH mission. Risk intelligence
capabilities can be measured along dimensions of governance and risk
oversight, risk identification, risk assessment, risk response,
monitoring and escalation, control assurance and testing, risk
intelligence performance and training, and sustainable and continuous
process improvement.

This section will outline and summarize the various data components of
the study to address the design of a methodology to assess NIH risks to
performance. These components include:

-   **Risk Assessment Perspectives from the Social Sciences:** A review
    of risk and risk-related perspectives from economics, social
    psychology, the psychology of decision making, and political
    science, with insights for consideration in the development of a
    comprehensive risk assessment methodology.

-   **Standards and Guidance for Risk Assessment:** Common set of
    dimensions and considerations that should be observed in a
    methodology that embodies risk intelligent process and product based
    on existing standards and guidance for risk management.

-   **Preliminary Observations from an ERM Survey:** Results from
    preliminary inquiries of organizations regarding the current state
    of the enterprise risk management practices in support of
    organizational risk intelligence and their implications for
    developing a risk assessment methodology.

-   **Recent ERM Case Studies:** Case studies of three organizations
    that have been are or are embarking on the implementation of
    enterprise risk management methodology and process.

### 3.2 Assessment Perspectives from the Social Sciences

Assessment is a component of decision models.[^17] As such its rules and
procedures are informed by the terminology, semantic relations, and
objectives of the assessment, the relationship of assessment to
assessors and assessees, and the objects of the assessment.

Since Enterprise Risk Management is a relatively new management
discipline, it is prudent to incorporate multiple perspectives in
formulating an initial methodology. Here are considered high level
summary observations from several management and management-related
disciplines, including psychology of decision making, political science,
sociology, social psychology, and economics.

### Political Science and Organizational Behavior Observations

How people contribute to the common good of a group, such as an
organization, is influenced by various contexts, including the framing
of the participation (benefits, sanctions, assignments, authorities, and
responsibilities), the relative scarcity of resources, predictability of
the organizational environment, the size of the group. The contribution
occurs in the context of a dialogue about risk between individuals and
the organization. The risk dialogue requires both candor, reflecting the
potential contribution of participants and closure, reflecting the
organizational discipline to make and follow through with
decisions.[^18]

Risk culture is the process which generates and sustains shared values
about risk. Perceptions, criteria, and assignment of risks can
Perception of risks may be amplified or attenuated by a range of
organizational and social processes. Given perception, risk messages are
sent through widening circles of organizational and social networks, and
fed back through discourse to local respondents.[^19]

Organizational trust in the sender of a risk message derives from two
sources: the fairness, consistency, competency of the sender; and
congruence of basic values about risk shared by sender and receiver of
risk messages. There are two feedback loops: one to clarify questions of
understanding; the other to discover shared values. Discovery of values
is often bound up and implicit in questions to clarify risk
understanding. In any case, judgments about risks, and what to do about
them, appear to reflect broader organizational stances on often highly
politicized issues. Follow-through by leadership contributes to a
climate of trust.[^20]

Summary of Insights:

-   Leaders set the tone about risk, its assessment, and its mitigation,
    always in the context of the decision making process itself:
    "leaders get the behavior they tolerate." Decisions are to be
    aligned with responsibilities and evaluated according to accountable
    performance.

-   Risk discussions should be embedded in regularly occurring
    management process: the information needed often exists in the
    organization, or at least someone knows how to get it

-   Group bias cannot be eliminated, but, as a risk in itself, can be
    mitigated through *tone at the top*, incentives, demonstration of
    "one team -- one fight" successes, and as part of the regular agenda
    of organizational discourse.

-   Make information sharing, especially about risks, the default.

-   Eliminate the risk extortionists (that hold the group hostage to a
    point of view), the risk side-trackers (those who recount history
    and the glory years), the silent risk liars (especially those who
    agree to actions they have no intention of doing), the risk dividers
    (those who create breaches by building parallel and unofficial
    decision making processes)

### Observations from Decision Making Psychology

Most risky decisions are made by people using simple approaches to
prioritization, often fraught with bias. The goal, often latent, is to
economize on the amount of time and effort needed to make a decision.
Some use a lexicographic approach so that the most important attribute
guides the decision maker to the most attractive alternative. Others
choose an alternative that meets the largest number of important
criteria. Still others may choose an action that meets a minimum
satisfactory threshold on decision criteria. Lay people and experienced
researchers employ stereotypes, reference to memorable experiences, and
adjust final estimates to meet their initial expectations, and rely on
decision's history to make a choice. Choice of decision formulation also
influences choices. For example, people who frame a decision in terms of
positive upside scenarios tend to be more averse to risk; while those
who adopt a loss frame tend to seek more risk, perhaps
paradoxically.[^21]

When faced with considerable change, people tend to protect their
hard-earned esteem. People tend to sub-consciously decide what to do
before they figure out how to do it. Add to this the tendency to want to
do what we like to do, and the result is that our decisions about very
negative issues, such as risk, are drawn to our tendencies. Similarly,
people make decisions based on their history with the issue. If they
invest resources, time, effort and have made irreversible commitments,
they will not abandon these "sunk costs" in favor of a new course of
action. A consequence of this behavior is the extra, and often
unsubstantiated, weight given by decision makers to the first bit of
information received. This can have the effect of "anchoring" a
decision, especially one that is important or has a high degree of
uncertainty associated with it, to that first wave of information. Also
related is the tendency to be radically over or under confident in
estimates about future outcomes.[^22]

The problem of communicating risks across different groups and with
different decision makers is whether sufficient information exists, and
if the decision makers can make good use of the information. The key is
to understand the mental models (assumptions, patterns, misconceptions)
that people bring to a deliberation. Experts possess good knowledge of
risks, but often mis-communicate results to non-experts, here called
"lay." Lay people will use their incumbent understanding of a risk to
deliberate on mitigating actions, often misinterpreting the expert's
opinion, which may not have been appropriately communicated in the first
place. Lay attitudes toward risk seem to depend on the degree of
"perceived dread." Thus expert opinions about risk may or may not carry
the intended message, as the expert's context differs in substantive
ways from the non-expert. be formed in cultural discourse, as much as
excavated from personal (or expert) experience and analysis. The
evidence continues to accumulate that lay decision makers may not always
trust or even rely on the opinions of experts.[^23]

Summary of Insights:

-   Reduce bias in decision making processes by training to look for
    risks, challenge the absence of opinion (internal and external; for
    or against a decision), when it is important overdo the analysis,
    consciously keep in mind the big picture.

-   View important risks from multiple perspectives against the context
    of the organization's "big picture."

-   Doing nothing, that is living with the status quo, is a decision
    with unknown, and often surprising and unintended, consequences

-   Review the quality of the decision process, not just the quality of
    the decision outcome. Good decisions (process) can lead to bad
    outcomes -- thus build into every decision an exit to minimize loss
    impact.

-   In every decision, especially those related to highly uncertain
    outcomes, consider the possibility of failure and build mitigations
    given the possibility of failure directly into mitigation plans.

-   Reframe choices to keep the "odds" out of the analysis. Instead of
    framing a choice as a loss with odds of gain or loss, or as a gain,
    with odds of gain or loss, restate the risk as a 50-50 chance of
    winning and losing outcomes.

### Observations from Experimental Economics and Finance

The focus of economics models is on the outcomes of a representative
decision maker in a market or quasi-market setting, not the process of
decision making. These model often require assumptions about the
decision maker which are not empirically viable, including that persons
make choices based on relative expected marginal utility. In turn, this
assumption requires that decision makers have a well-ordered approach to
prioritizing outcomes of decisions, know enough about the future to
forecast it consistently with the outcomes of decisions, and can lay off
undesirable outcomes in well-ordered and trusted markets. Under rational
expectations, the decision maker is assumed to simultaneously predict
the future while making a decision that affects future outcomes. If the
decision maker is influential enough, the prediction and the decision
are much the same, resulting in a logically self-fulfilling
prophesy.[^24]

The current economics of risk separates outcomes into risky outcomes
(known probabilities) and uncertain outcomes (those outcomes that might
be inferred but otherwise not calculable). However, the economics
approach starts with a representative decision maker from whom
calculable probabilities can be derived. Since economists use past data
to calculate raw or inferred probabilities, full knowledge about the
future is not available, and thus the future outcomes of a decision
remain uncertain -- a caveat of the economic model of risk.[^25]

Summary of Insights:

-   Use economics models to assist decision makers understand the logic
    of risk-based decisions and to develop representations of the
    consequences of financial impacts.

-   Use historical data to develop a baseline understanding of the past
    consequences of decisions, where available, and to project potential
    outcome scenarios conditional on baseline assumptions

-   Use economics models to simulate the impact of shocks on uncertain
    outcomes

-   Develop risk scenario approaches that allow for the effect of
    changes in decision maker preferences and choices

-   Communicate risks in a way that does not depend on probabilistic
    statements, especially those based on history.

### 3.3 Standards and Guidance for Risk Assessment

A critical capability, and key entity level control[^26], is an
organization's process for *risk intelligence*.[^27] The risk
intelligent manager (whether executive officer, business unit leader, or
process owner) will deploy an Enterprise Risk Assessment (herein
referred to as "*Risk Assessment*") process that encompasses an
organization's key risks to achieving stakeholder value. The basis for
the organization's planned response to risk is derived from a
fundamental result of the assessment, the amount and location of the
organization's mitigated value. The risk response plan must at least
assure stakeholders and executives of the existence and sufficiency of
an organization's mitigated value on a risk-prioritized basis.

The content of a methodology to assess risk includes at a high level the
organization's business risks. These may be categorized[^28] as:

-   **Governance Risks -** Risks related to the structure, policies,
    procedures and authorities in which the key directions and decisions
    of the organization are overseen.

-   **Strategy & Execution Risks -** Risks associated with an
    organization's inability to formulate and/or execute a successful
    business strategy largely having to do with the future growth plans
    of the organization such as plans to enter new markets, launch new
    products, and form new alliances.

-   **Operational Risks -** Risks arising from inadequate risk controls
    or failure of risk infrastructure largely having to do with the
    protection and utilization of existing assets and how they may be
    leveraged for future mission objectives. Related to breakdowns in
    the performance of people, processes and systems that support the
    operations of the organization.

-   **Infrastructure Risks -** Risks related to breakdowns in the
    performance of people, processes and systems that support the
    operations of the organization.

-   **External Risks -** Risks associated with factors external to the
    organization, whose likelihood cannot be controlled by the
    organization.

These risks are related directly to the ability of an organization to
meet its value objectives as listed in an organization's performance
objectives,[^29] for example:

-   Revenue growth: where the organization may failure to meet customer,
    product, market goals

-   Margin: where the organization may fail to meet cost targets,
    including restructuring of costs, and provision of services

-   Assets: where the organization may fail to meet asset turnover,
    flexibility, effectiveness and efficiency targets

-   Expectations: where the organization may fail to meet various
    stakeholder expectations, including shareholders, regulators, rating
    agencies, banks, employees, customers and suppliers.

These financially related goals have analogues for public sector
organizations[^30]:

-   Policy Directives: where the organization may failure to meet
    customer, product, market goals

-   Program Delivery: where the organization may fail to meet program
    requirements on time and within budget

-   Operational Efficiency and Asset Effectiveness: where the
    organization may fail to meet asset applicability, flexibility,
    effectiveness and efficiency targets

-   Expectations: where the organization may fail to meet various
    stakeholder expectations, including legislatures, constituents,
    regulators, rating agencies, employees, customers and suppliers,
    other public and private sector organizations.

The risk assessment links risks to value by answering the following
questions:

1.  How can the organization fail to achieve its objectives?

2.  What would cause the organization to fail?

3.  What would be the effects of the failure?

4.  What is currently being done to prevent, detect, correct or escalate
    such failure?

5.  What is the organization's vulnerability to such failure?

6.  What further actions are required to cost-effectively mitigate
    failure?

7.  How does the organization get reasonable data assurance that
    existing mitigation is reliable and effective?

These questions form the basis of the competency of the methodology to
meet the goals of risk intelligence. It is critical to use the Risk
Assessment as a management and audit tool that recognizes that the
organization should take only rewarded risk and avoid, or even
eliminate, unrewarded risks. The results of a Risk Assessment can be
used as a risk management tool to present a point-in-time evaluation of
the organization's risk profile and can be a starting point to launch a
formalized enterprise risk management process that includes the
designation of risk owners/managers and a process for on-going risk
management monitoring and reporting. The existence and effectiveness of
a risk assessment process itself is an organization level control.

The risk assessment process can be leveraged to support risk response
planning and enhancement programs in several ways according to several
standards and guidance:

-   **Impact / Materiality**: Provides a consistent means for focusing
    risk response and review efforts on processes, systems, entities,
    and accounts with the greatest affect on the achievement of business
    objectives including strategy, operations, reporting and compliance
    (e.g., as indicated in COSO II).

-   **Shock Analysis**: Allows modeling of risk scenarios, including
    correlations and domino effects, to better gauge aggregated impact
    on key performance metrics.

-   **Failure modes**: Identifies potential breakdowns in performance,
    processes, people, systems and assets that increase vulnerability of
    failure to achieve expectations.

-   **Preparedness and Vulnerability**: Links risks to risk response
    objectives, activities and other risk mitigation measures to
    determine management's ability to detect, prevent and correct
    unfavorable events and conditions.

-   **Mitigated Value**: identifies the level of mitigation achieved or
    desired as the difference between impact and vulnerability.

-   **Assurance**: Identifies areas where increased or sustained
    assurance efforts are needed especially for high levels of mitigated
    value.

-   **Efficient Deployment of Resources**: Provides a basis for
    allocating limited resources for compliance testing, remediation,
    and management action.

Management must assure itself that it is sufficiently mitigating risks
it is authorized to take, eliminate risks, and where possible, that it
is not authorized to take and that are intolerable, and take preventive,
detective, corrective action around risks that exceed the organization's
risk appetite. From the point of view of organization assumption of
risk, risk assessment will review management's answers to the following
questions:

1.  What are the policy and guidelines for assessing and managing risks?

2.  What are the organization's key risks and vulnerabilities and the
    plans to address them?

3.  What is the organization's risk appetite and how much risk has the
    organization taken on?

4.  Who has the responsibility and authority to take risk on behalf of
    the organization?

5.  What is the organization's capability to manage risk on an
    integrated and sustainable basis?

These questions are used to develop the 8 dimensions of risk
intelligence and management capability and thus summarize practices from
diverse guidance and standards. Using the data analysis procedures
outline in the Data Analysis section of this study, 8 dimensions of risk
intelligence are developed.

The 8 dimensions of risk intelligence can then be used as a tool to
harmonize, synchronize, and rationalize an organization's risk-related
governance, compliance and management processes. Each dimension is a
process associated with the risk management cycle of activities and
linked to the organization's existing management decision processes.

The eight dimensions that emerged by applying the study approach to
developing this tool are:

1.  Governance and Risk Oversight: This encompasses the organization\'s
    tone at the top, culture, risk governance structure (e.g.,
    committees, charters, and authorities), risk management policies
    including tolerance for specific types of risk (i.e., whether or not
    it is willing to take certain risks) and appetite for those risks
    the organization is willing to take. This process is the basis for
    all other components of enterprise risk management and risk
    intelligent decision-making including its philosophy about how risk
    should be understood and managed. Authority and direction is
    exercised by properly designated managers over assigned resources in
    the accomplishment of business objectives.

2.  Risk Identification: Management identifies potential internal and
    external events that are relevant to the business and could
    significantly affect the entity. Risks to objectives should be
    considered as scenarios and chains of events rather than as isolated
    incidents. This includes risks to future growth objectives (rewarded
    risks) as well as risks to existing assets (unrewarded risks).

3.  Risk Assessment and Measurement: Risk assessment enables the
    business to consider the extent to which potential events may have
    an impact on achievement of objectives, the residual exposure of the
    business after taking into account current risk mitigation and
    controls and to prioritize the allocation of resources. The
    assessment uses inherent and residual risk to prioritize risks to
    the achievement of an organization's goals and a rating of the
    capability of the organization to manage its mission critical risks.
    The difference between gross and net risk is the level of mitigated
    value. Measurement of gross and net risk can be carried out
    qualitatively (as in categories from very low to very high, e.g.,
    employee satisfaction) or quantitatively (as in levels of a
    performance metric, e.g., number of grants requiring rework). There
    are two components of a gross risk measurement: level of loss
    emanating from a threat and speed of onset of the loss. Similarly,
    there are two components of a net risk: the degree of residual loss
    or vulnerability to a threat and speed of response by management to
    the threat.

4.  Risk Response Alternatives and Plans: Risk response is management\'s
    determination on how to respond based on its assessment of the
    relevant risks. This includes whether to avoid a risk, accept it or
    transfer it. Risk response mechanisms include preparation for,
    response to, and recovery from risk events as they unfold. Action
    plans include management mitigation, assurance, redeployment and
    cumulative impact. Examples of management action include prevention,
    detection, correction and escalation of risks as embedded in
    internal controls and risk response plans. Plans may be further
    prioritized according to mitigation, assurance, redeployment and
    measurement for cumulative impact.

5.  Monitoring and Escalation: Monitoring is the periodic or continuous
    observation of the enterprise's portfolio of risk exposures in order
    to detect and give timely warning of change. Escalation includes
    procedures in which risks that exceed thresholds or triggers are
    elevated to the appropriate level of authority.

6.  Control Activities, Assurance, and Testing: Control activities are
    the policies and procedures that help ensure that management's risk
    responses are carried out. Control activities occur throughout the
    organization, at all levels and in all functions. They include a
    range of activities − as diverse as approvals, authorizations,
    verifications, reconciliations, reviews of operating performance,
    security of assets, and segregation of duties. Assurance activities
    include an objective examination of evidence for the purpose of
    providing an independent assessment on risk management, control, or
    governance processes for the organization.

7.  Risk Intelligence, Communications, and Training: Risk intelligence
    results from collection, processing, integration, analysis,
    evaluation, and interpretation of available information concerning
    risks to the enterprise obtained through observation, analysis, and
    understanding. Pertinent risk intelligence is identified, captured,
    and communicated in a form and timeframe that enables trained people
    to carry out their responsibilities. Personnel are trained to make
    rapid and appropriate decisions using risk intelligence provided to
    them.

8.  Sustainability and Continuous Improvement: Risk intelligence should
    be maintained indefinitely and depends on the capability of people,
    processes, systems, and assets to act in a harmonized, synchronized,
    and rationalized manner. Continuous improvement is based on the
    assumption that further improvements are always possible and that
    risk management processes should be continuously reevaluated and
    prioritized improvements implemented.

Within these 8 processes, there are 89 sub-categories based on global
leading practices that incorporate and integrate for example, COSO ERM,
NYSE, EuroNext, Australia New Zealand, KonTraG (Germany), Turnbull
(U.K.), ISO14000, King (South Africa), Dey (Canada), and other risk
management standards, guidance, and practice. Each sub-category contains
entity level considerations to enable assessment of current state versus
desired state; tests of capability; and appropriateness of design,
effectiveness, efficiency and responsibility.

A table with the 8 dimensions, operational definitions, and
sub-dimensions is in the Appendix. The 8 dimensions is the comprehensive
list of standards and guidance that are used to develop the risk
assessment design. As a group, they provisionally answer the first study
question regarding the "best," that is, at least the leading approaches
available to the design of a methodology. In the context of this study,
these are better classified as "leading" practices in that they are
incorporated into standards and guidance for consideration by
organizations.

### 3.4 Examples of Risk Assessment Methodologies

Examples of enterprise risk assessment and management methodologies are
provided here for three reasons:

-   To illustrate the level of active interest of large, complex
    organizations in developing, deploying, and maintaining a unified
    approach to risk management scalable to diverse organizations

-   To indicate that a body of public sector leading practice is
    accessible as a reference for similar implementations

-   To illustrate components of the 8 dimensions of risk intelligence

The three examples include the Joint Standards Australia/Standards New
Zealand Committee OB-007 issuance of the Risk Management standard, Great
Britain, H.M. Treasury issuance of the Green and Orange Books, and
National Institute of Standards and Technology Special Publication
800-30 along with OCTAVE(sm) (by Carnegie-Mellon University, Software
Engineering Institute). Each example applies to large scale public
sector organizations, is maintained as guidance for specific
implementation of an organization's program, and provides detailed
practices for potential use by an organization.

The Australia New Zealand Standard: Risk Management (AS/NZS 4360:2004)
was prepared by the Joint Standards Australia/Standards New Zealand
Committee OB-007, Risk Management. The standard provides "a generic
framework for establishing the context, identifying, analyzing,
evaluating, treating, monitoring and communicating risk." The standard
incorporates by reference several other diverse standards, including for
example, Australian versions of ISO 9000 Quality management systems---
Fundamentals and vocabulary, ISO 14004 Environmental management
systems--- General guidelines on principals, systems and supporting
techniques, ISO 14050 Environmental management---Vocabulary, ISO 15489
Records management.

Great Britain, H.M. Treasury has issued the Green Book to provide a
common methodology for central government appraisal and evaluation, and
the Orange Book to provide risk management principles and conceptual
guidance. The Green Book considers risk assessment as a mission
critical[^31] component of management: "a structured approach to
identifying, assessing and controlling risks that emerge during the
course of the policy, programme or project lifecycle." Its purpose is to
support better decision-making through understanding the risks inherent
in a proposal and their likely impact. In their view, risk assessment
and management support change management, resource allocation, and the
minimization of fraud and waste.

The Green Book also advises consideration of the following themes for
governance and risk oversight:

-   Institute a risk management framework, within which risks are
    identified and managed;

-   Senior management must support, own and lead risk management
    policies;

-   Senior management must clearly communicate organizational risk
    management policies to all staff;

-   Risk management must be fully embedded into business processes and
    ensuring it applies consistently;

-   Establish an organisational culture that supports well thought out
    risk taking and innovation.

Components and considerations included in Orange book risk assessment
methodology include:

-   Senior management determines which risks are tolerable, and for
    tolerable risks, the organization's appetite for risk

-   Residual risk (impact on organization performance against mission
    with consideration of management risk response and internal control)
    is insufficient to prioritize risk; needed is consideration of
    inherent risk (impact without consideration of management response):
    the extent to which the risk should be addressed by senior
    management is a consideration of inherent risk, while the means to
    respond to risk is considered by residual risk

-   Documentation of continuous risk assessment enables the organization
    to view overall risk and how the responsibility is assigned,
    accounted for, controlled, and monitored

-   The highest priority risks should be regularly considered by the
    highest level of organizational governance.

The Federal Information Processing Standards("FIPS") of the National
Institute of Standards and Technology ("NIST") includes guidelines
adopted and promulgated under the provisions of Section 5131 of the
Information Technology Management Reform Act of 1996 (Public Law
104-106) and the Federal Information Security Management Act of 2002
(Public Law 107-347). These mandates have given the Secretary of
Commerce and NIST important responsibilities for improving the
utilization and management of computer and related telecommunications
systems in the federal government, an example of a pervasive, mission
critical, "enterprise-wide" process and system. The NIST, through its
Information Technology Laboratory, provides leadership, technical
guidance, and coordination of government efforts in the development of
standards and guidelines in these areas. An example of a risk-based
standard is FIPS 199 - Standards for Security Categorization of Federal
Information and Information Systems. Implementation of this standard
through OMB Circular A-130 and A-4 and NIST Special Publication 800-30
(2002) among others.[^32] This guidance is included as an example for
the following reasons:

-   It is guidance for an enterprise-wide risk (security of IT assets)

-   It is directly linked to mission critical objectives and activities
    of agencies

-   It is required under GPRA (systems to support budget preparation and
    reporting) and FISMA

This standard and supporting guidance include:

-   Risk management must be enable the organization to accomplish its
    mission(s) "(1) by better securing the IT systems that store,
    process, or transmit organizational information; (2) by enabling
    management to make well-informed risk management decisions to
    justify the expenditures that are part of an IT budget; and (3) by
    assisting management in authorizing (or accrediting) the IT systems
    on the basis of the supporting documentation resulting from the
    performance of risk management." (SP 800-30)

-   Inclusion of both physical and administrative impact attributes
    (e.g., assessment and categorization of the campus power plant SCADA
    system) in an assessment for potential loss (FIPS 199)

Embedded in SP 800-30 are elements of Carnegie Mellon University,
Software Engineering Institute tool called OCTAVE: Operationally
Critical Threat, Asset, and Vulnerability Evaluation. In the OCTAVE
environment, risk owners are responsible for the evaluation and
management of risk. They are aided by an analysis team to challenge and
bring additional insight and rigor into the evaluation of assets,
threats and vulnerability. In this way OCTAVE brings appropriate levels
of resources and responsibilities to the focus on those assets that are
most critical to an organization. The OCTAVE approach emphasizes the
role of organizational elements (evaluators who are risk owners versus
analysts who facilitate) in building threat profiles, identifying
infrastructure vulnerabilities and developing risk strategies and plans.

##     

## 3.5 Preliminary Observations from an ERM Survey

This section summarizes the results of an ERM web-based survey
administered to U.S. and European companies over the past 12 months.
Most of the companies are in consumer business, energy, manufacturing
and financial service industries. They are selling more than \$5 billion
per year of goods and services. Respondents include general auditors,
general counsels, chief executives, chief financial officers, chief
information offices, and others designated by these executives. Most of
the companies are publicly listed. Over 20% reported a significant loss
during the past five years. Many report an increased interest in ERM
relative to one year ago.

![](media/image3.wmf)![](media/image4.wmf)![](media/image5.wmf)![](media/image6.wmf)![](media/image7.wmf)![](media/image8.wmf)
![](media/image9.wmf)![](media/image10.wmf)

Regulation continues to drive interest in ERM. This factor is followed
by reputation and image and then by stakeholder expectations. Regulation
and reputation are two aspects of management that tend to run through
all strategies, operations, governance and infrastructure. Both can be
the source of shocks that critically impair the ability of any
organization to meet its mission objectives. The implication for
methodology is to identify, assess, respond to, and monitor all mission
critical risks that can possibly be linked to pervasive outcomes such as
reputation and compliance.

![](media/image11.wmf)![](media/image12.wmf)

Senior management, the Board, internal audit all contribute to the
degree of higher interest in ERM. As a result many companies report at
least the development of formal ERM programs, with most in existence for
2 years or less.

![](media/image13.wmf)![](media/image14.wmf)

Some companies are already experiencing the benefits of a risk aware
culture, sustenance of stakeholder trust, reduced vulnerability to
adverse events, and enhanced risk response. Most companies anticipate
these benefits and more, such as minimizing surprises, and providing
integrated responses to multiple risks. Such benefits should be
anticipated by an ERM methodology.

![](media/image15.wmf)![](media/image16.wmf)

Most respondents see the role of the "ERM function" as facilitator and
recommender, not as manager. The primary goals of an ERM program are
governance, transparent reporting, alignment of operations with strategy
and objectives.

![](media/image17.wmf)

Where is ERM integrated? Most respondents believe ERM is integrated with
compliance, operations, security, treasury, controller functions, and
ethics. ERM does not seem to be integrated with two critically important
functions, marketing where the organization's brand becomes public, and
human resources. Methodology must seek to include all critical aspects
of the process of operating an organization.

![](media/image18.wmf)

There are a variety of organizational models for the risk function. They
all have this in common: the risk function facilitates the
identification, assessment, response, monitoring and escalation of
risks; business units and the chief executive assess, respond to, and
executive risk response plans. Nearly one third of respondents indicate
that there is no systematic function in their organizations for managing
enterprise risk. In interviews with executives, such organizations
assume it is the role of the chief executive. But in the same interviews
it is often revealed that the chief executive does not systematically
incorporate risk into the decision making process. One implication for
methodology is develop a risk facilitation function at the top of the
organization, responsible for the conduct of the risk assessment
process, its continuous improvement, and sustainable embedding in other
risk-related processes, such as compliance and ethics.

![](media/image19.wmf)

Most chief risk officers, or the function associated with this title,
report to the traditional risk facilitator in an organization, the CFO.
In most organizations, it is made clear that the chief executive is the
chief risk executive as well, executing risk response plans. The CFO and
the CRO facilitate the discussion, assessment, and reporting of risk in
an exhibition of segregation of duties. Stakeholders can then assure
themselves that a check and balance is maintained so that risk response
can be appropriately challenged as decisions are made.

![](media/image20.wmf)

Most organizations have established a risk management oversight function
and committee. Methodology is effective only with complete support of
the executive and oversight functions of an organization. A good
methodology directly incorporates oversight into the prioritization and
resource allocation process.

![](media/image21.wmf)

Most have implemented policies, are conducting a periodic
organization-wide assessment, building scorecards and dash boards. It is
surprising the percentage of respondents who are not charging one
committee to deal with all risks, who are not attempting to measure risk
against risk appetite, or integrating risk into other management
functions. The implication for methodology is to focus on these areas to
provide top-down governance and guidance, use data-driven assessment and
measurement against established limits.

![](media/image22.wmf)

Respondents indicate that a wide variety of risks are identified. ERM
programs tend to focus on business continuity, legal liability,
integrity and ethics, controls, various silos. However, possibly a small
sample aberration, mission critical risks such as incentives, mergers
and acquisitions, and privacy are not typical in focus. The implication
for methodology is not to typecast ERM as the "fix" for obvious
operational risks, but to include governance, strategy, operations,
infrastructure and external factors into one holistic and harmonized
approach.

![](media/image23.wmf)![](media/image24.wmf)

Respondents indicate that both inherent (gross) and residual (net) risk
are the primary criteria used to assess risk. They also indicate that
probability is a prevailing practice. However, in the next question,
they indicate that they have medium to low confidence in ability of
likelihood to predict loss. Mission critical losses rarely happen,
statistically. However, when they do, they can, precisely because they
are mission critical, significantly impair the ability of an
organization to withstand a major shock. In interviews with executives a
recurring theme is that probability (or colloquially, likelihood) was
inherently unable to help them predict any major loss. In fact, one
executive exclaimed that the only thing predictable about likelihood is
its lack of predictability of mission critical risks. The implication
for methodology is to incorporate both inherent risk and residual risk,
and if one were to use probability, use it appropriately. In several
disaster risk management systems there is use of onset or speed of onset
to attempt to modify the size of a loss. This factor can be scored and
calibrated using facilitated assessment techniques. The factor can
indicate the suddenness or gradualness of the approach of a threat. In
any case, most respondents assess enterprise risk at least on an annual
basis.

![](media/image25.wmf)

One of the hallmarks of good decision making is the ability of a
methodology to drive the "noise" out of the system. Most mission
critical decisions happen in strategy, capital allocation, mergers and
acquisitions, through delegations of authority, and in areas of
compliance, and where health and safety are paramount. Respondents
indicate that these areas have not yet fully incorporated risk in a
systematic fashion into their decision processes. The challenge and
implication for methodology is to design an approach that shares common
ground with existing executive decision process, controls, reports, and
practices.

![](media/image26.wmf)

This and the next question go to the heart of enterprise risk
management. This question asks whether the organization is prepared to
deal with mission critical risks, those risk, should they occur, would
significantly impair the ability of the organization to meet its
objectives. These respondents generally agreed that, yes, they are
prepared.

![](media/image27.wmf)

However, when asked about the degree of confidence in the effectiveness
of risk response, about half of the respondents indicate that their
organizations have medium to low confidence. The implication for
methodology is to focus on these risks, which in turn requires an
approach that can effectively gather diverse elements of complex
organizations, drive consensus to identify the list of mission critical
risks, and then prioritize them according, all on a regularly occurring
basis. But that implication is insufficient. If organizations believe
they have mitigated a mission critical threat, they must then assure
themselves in their belief that the mitigation is sufficient, effective
and efficient. If the mitigation fails to meet risk tolerance, or if
tolerable, risk appetite standards, then further corrective action may
be required. The task of assurance in that case is to monitor the
process of corrective action planning.

![](media/image28.wmf)

Many participants report the use of standard risk measurement and
monitoring tools. It has become standard to see reports with scenarios
and sensitivity analysis and quality tools such as root-cause and
failure modes and effects analysis. This is in part due to the ongoing
development of quality management approaches in many organizations. It
is also indicative that the most challenge comes from the use of
probabilistic and more complicated performance metric tools. In
interviews, executives emphasize the need to keep risk, and performance,
measurement simple, easy to disseminate and communicate, and indicative
of relative priorities.

![](media/image29.wmf)

With these two questions, respondents are asked to comment on the
existence of risk tolerance and appetite. Here respondents were
consistent in that they reported at least partial identification of
intolerable risks, and for those risks that are tolerable, an assigned
appetite. The implication for methodology is to systematically align
tolerance with appetite.

![](media/image30.wmf)

Given an alignment of tolerance and appetite, respondents consistently
answered these two related questions about monitoring and escalation,
both of which are recommended in standards and guidance. The question
that would follow these is a focus on "partial" responses. It would
attempt to understand the focus of partial monitoring and escalation
efforts. Are the efforts focused on critical risks? Is escalation
defined early enough to avoid a threat incidence? In interviews,
executives either did not know, or suggested that certain mission
critical risks are regularly monitored and escalated, often in line with
an emergency response plan and process associated, for example, with an
integrity breach. The implication for methodology is to build a
straight-through process from identification of risks, assignment of
tolerances and appetite, on to monitoring and escalation. The escalation
function ties the risk process back to governance, responsibility,
accountability, and the organization's competency to manage a specific
risk.

![](media/image31.wmf)

This question addresses the dissemination of risk capability among "lay"
risk employees. The respondents reveal a preference for risk specialists
and thus the notion that risk silos are currently sufficient to generate
risk information and manage risks. In interviews, executives continue to
voice concerns that "risk silos" are not able, or in some cases, willing
to communicate with one another. Similar to quality, risk is experienced
and acted on by all management and staff in an organization. Thus a
methodology should seek to perform two tasks: preserve the integrity of
silos, for that is where deep knowledge of specific risks and their
management resides; enhance communication among silos, to provide
effective cross-functional management, learning, and coverage.

![](media/image32.wmf)

The purpose of this question is to begin to understand the degree of
communication with the "Board" that oversight body fundamentally charged
with overseeing risk in an organization. Respondents indicate a regular
flow of risk information to the Board. This may mean that risk
information is required by the Board, that management proactively offers
risk information to the Board or a combination of the two. With such
regularity of flow, an effective and efficient process for producing
risk information to the Board is needed, and should be built into the
design of a methodology.

![](media/image33.wmf)

A methodology should exhibit ability to improve continuously and sustain
change to the organization. In the early stages of a new approach as
with ERM, it is useful to gather perceptions about significant
challenges facing implementers. Knowledge of these challenges enables
formulation of methodology that can avoid known vulnerabilities and
enhance other areas for a phased improvement. In the survey, respondents
indicate high degrees of challenge emanating from risk assessment and
measurement, proving the business case, and understanding the
organizational benefits of risk management. This underscores the strong
need in these organizations for improved communications, education,
clear and concise benefits to decision makers, and linkage to
governance, performance and decision making process. Designing these
elements into a methodology will help eliminate inefficiency and
ineffectiveness of the risk process, especially at points where the risk
process embeds into mission critical processes such as strategic
budgeting and emergency response.

![](media/image34.wmf)

In this question, organizations are asked to comment on the tools they
currently use to manage the enterprise risk process. This question
strikes at the maturity of organizations surveyed. Typically,
organizations will first institute risk governance, lay out roles and
responsibilities, establish a risk process. After those initial steps,
organizations will then embed the risk process into regularly occurring
governance and decision processes, begin to provide motivation in the
form of accountabilities and incentives, and start to convert
predominantly manual process to more automated platforms. A more telling
result is the relative immaturity of respondents in terms of linking
risk to performance. A caution to any development of methodology is
first get the process right, then specify business requirements, finally
developing technological specifications. First pilot the use of a
process before trying to impose the specific standardizations implied in
automation.

During the course of the survey, interviews were held several
executives, Board members, senior advisors and managers. Included here
are summaries of implications for ERM implementation based on the survey
and interview results and organized by interview question:

**Why is integrated enterprise risk management (ERM) such an important
issue these days?\
**First, corporate risk has become personal. Boards and executives are
very concerned about personal liability and are demanding greater
transparency and assurance of due diligence, due care and good business
judgment. 

Second, risk is managed within silos in some organizations today and it
is not integrated or coordinated. The problem is that risk doesn't
respect those silos. Eighty percent of all major value losses involve
the interaction of more than one risk. So, companies may find themselves
completely unprepared to face the kinds of interactions of adverse
events that can threaten their success or even their survival.

Third, much of risk management to date has focused on the protection of
existing assets and adverse events that can affect those assets. This
approach does not distinguish between two types of risks: unrewarded and
rewarded risks. Unrewarded risks are typically associated with financial
reporting and compliance and operations. This is the traditional domain
of risk management because it focuses on the protection of existing
assets. Organizations aren't rewarded for volatility in their financial
performance or manipulating their financial reports. They can lose 50
percent of their market capitalization in forty-eight hours or less if
they have an integrity-related issue. They don't get rewarded for being
noncompliant.

**Implications:**

-   Where possible use incentives and other motivations such as
    performance contract elements to make risk personal to those
    accountable for risk response

-   Risks do not appear in single events, but act together, so build
    interactivity of risks and scenario capability into risk
    identification

-   Do not fail to review risks to future mission objectives

**What types of risks are rewarded?\
**Rewarded risks have to do with the big bets organizations place in
terms of strategy and its execution. Organizations are rewarded for
successfully taking risk that offers them a \[competitive\] advantage.
The biggest risk one may have is not taking one. Organizations that fail
to keep pace, fail to challenge their business models, fail to enter new
markets, fail to successfully develop and launch new products are going
to be passed up by companies who take those calculated risks. Strategy
is all about making profitable bets. Organizations need a systematic way
of understanding what makes a profitable bet versus an unprofitable bet.

**Implications:**

-   Focus risk assessments on mission critical risks, those risks that
    result from placing assets, people, capital at extreme risk

-   Some of the largest risks emanate from lack of strategy, so be sure
    to include strategic risks on the list of mission critical risks

**How is enterprise risk defined? Does it apply to specific operational
issues, financial risk, or to broader issues companies deal with?**

It deals with all risks to the organization ---from all sources. The
problem is that risk management in the past has been largely confined to
dealing with the risks to existing assets as opposed to future growth.

Integrated ERM is simply a set of tools that, if properly deployed, will
provide an organization with better risk intelligence not only about the
big bets it makes but also the risks to its existing asset base. An
integrated ERM system needs to provide two functions: a system and
discipline that will help the organization understand and manage
vulnerabilities related to a range of scenarios, and a discipline and
system that will build better risk intelligence into the way decisions
about future growth are made and then executed.

**Implications:**

-   Identify risks from all sources, so do not curtail the scope of risk
    assessment

-   Deploy risk assessment as an agent and occasion for changing
    organizational culture to manage mission critical vulnerabilities

**Why has there been a focus on operational issues?**\
Historically, risk management has been the domain of those people who
are charged with protecting existing assets. So when you come to talk to
about risk management, they want to focus on controls and driving down
the cost of good risk management without really understanding the costs
of poor risk management and the return on investment ("ROI") value of
better risk intelligence. Companies need different tools to address
adverse scenarios and events and tools that help make better
risk-informed decisions.  Both are required.

**Implications:**

-   Risk management is everyone's job, just like quality and integrity

-   Build and deploy risk intelligent decision making tools that
    provides data for making sound decisions accessible to all of the
    organization's decision makers

**What audience within the organization is most receptive to an
integrated approach to ERM?**\
Only the board and executive management are in the right position. A
bottom-up approach is likely to fail unless the board and executive
management agree that there is value to be obtained from better \[risk\]
intelligence.

**Implications:**

-   Critical to the success of any risk assessment and subsequent
    response and corrective action plan are senior management's support,
    virtual and material

**What is a reasonable approach to risk assessment?\
**Today, many organizations use a combination of impact and likelihood
to assess risk. This approach is potentially flawed. Over the course of
the last sixty months, the predictive value of likelihood for any of the
major catastrophes that have occurred was effectively zero. How likely
was September 11? How likely was Enron or WorldCom? Likelihood is not a
good predictor. A more reliable way is to agree on what scenarios and
events can really hurt the organization and whether or not we're
vulnerable to those scenarios. Vulnerability assessment is much more
meaningful to management in terms of what they need to prepare for.

**Implications:**

-   Most risk management disasters occurred at least partly as a result
    of assigning a near zero probability to extreme events, whereupon no
    management action was required

-   Use both inherent risk and residual risk to prioritize management
    action around risk scenarios

-   Use speed of onset to modify threat outcomes to measure inherent
    risk

-   Use speed of management response to modify degree of vulnerability
    to measure residual risk

-   Rank risks first by inherent risk, then by mitigated value, the
    difference between inherent and residual risk

-   Provide assurance for high mitigated value risks to processes and
    systems

-   Provide mitigation to very low mitigated value risks to processes
    and systems

-   For high residual risk and low inherent risk, consider further
    monitoring of the cumulative impact of risks to processes and
    systems

-   For low inherent and residual risks to processes and systems
    consider redeploying resources to higher risk areas

**What tools can help organizations manage risk?\
**Needed are a set of tools that will allow an organization, independent
of specific risks, to understand its stage of development and its
current capabilities in managing risks. This assessment helps
organization understand where they are relative to where they need to be
given the volatility of their environment. The first step is to identify
capability gaps and then close those gaps. The second step is to support
the organization in the development of better risk management
capabilities.  The third step is to build risk intelligence into the way
the organization makes decisions and does business.  In this way, the
organization can improve both the effectiveness and efficiency of the
way it understands and manages the risks and rewards of value creation
and the risks of value preservation.

**Implications:**

-   Phase risk management in successive waves into the organization,
    begin at the top of the organization, then, on a business unit by
    business unit basis, roll out risk identification, assessment, and
    response planning

-   Build incentives, such as gain sharing, to perform risk management

-   Embed risk management into regularly occurring management decision
    processes in order to reduce the burden on business operations and
    management

## 3.6 Case Examples of Recent ERA Development

Case studies and case-based reasoning are typically used when there is a
study environment that is not yet stable (e.g., new products, processes,
strategies), for which there is an evident paucity of usable data (e.g.,
one-off projects, organizations, and policies). Here are three summaries
of three case examples are used to illustrate the fact that formal
programs of ERM and ERA are being performed by complex, large
organizations. Each case summary answers these questions:

-   What is the business challenge that presented itself?

-   What was the solution selected by the organization?

-   What are the results of the solution?

-   What are the benefits of the solution?

-   What are the lessons learned by the organization?

In each case a formal request was made of management by an oversight
group (e.g., Board of Directors or Board of Trustees) to consider an
approach to manage risk on an enterprise-wide basis. Each organization
combined internal and external resources to scope, specify, design,
build and implement the program. Each organization operates the program
with internal resources.

### Case 1: Subsidiary of a Major Global Process Chemical Company

+-----------------------------------------------------------------------+
| **Business Challenge:**                                               |
+-----------------------------------------------------------------------+
| -   Current business environment is moving towards Enterprise Risk    |
|     Management and the first phase is Enterprise Risk Assessment.     |
+-----------------------------------------------------------------------+
| -   The firm required scoping, planning, data capture and data        |
|     analysis, facilitation, documentation, and reporting of           |
|     management's self-assessment of the major risks that              |
|     significantly impact the company's continental operations, and    |
|     the actions being taken to manage those risks.                    |
+-----------------------------------------------------------------------+
| **Solution:**                                                         |
+-----------------------------------------------------------------------+
| -   Orientation to selected staff in the use of the methodology and   |
|     associated tools.                                                 |
+-----------------------------------------------------------------------+
| -   Facilitated management's identification of selected mission       |
|     critical risks to the organization and their potential impact on  |
|     expected contributions to the parent's company shareholder value, |
|     by utilizing Integrated Enterprise Value at Risk (IntEVaR™).      |
+-----------------------------------------------------------------------+
| -   Facilitated assessment of the firm's current risk management      |
|     capabilities compared to the desired state of capability and      |
|     identify strengths, weaknesses and priority gaps.                 |
+-----------------------------------------------------------------------+
| **Results:**                                                          |
+-----------------------------------------------------------------------+
| -   Seamless transfer of knowledge, the assessment team was directly  |
|     involved in every aspect of the project.                          |
|                                                                       |
| -   The continuous "On the Job" direction and coaching throughout the |
|     project provided team members with first-hand knowledge and       |
|     applied learning that will assist them in sustaining the approach |
|     across all of firm's processes or functions.                      |
|                                                                       |
| -   2.5 Full Time Equivalent facilitation team members facilitate all |
|     risk assessment activities across business units                  |
|                                                                       |
| -   Risk Assessment embedded in compliance management and leadership  |
|     team process                                                      |
+-----------------------------------------------------------------------+
| Ongoing benefits are:                                                 |
|                                                                       |
| -   identification of key enterprise objectives and drivers           |
|                                                                       |
| -   identification of mission critical risks to the enterprise        |
|     objectives and drivers                                            |
|                                                                       |
| -   identification of executive risk owners and subject matter        |
|     specialists and assignment of accountability                      |
|                                                                       |
| -   risk assessment, analysis and reporting processes                 |
+-----------------------------------------------------------------------+
| **Organizational Learning:**                                          |
+-----------------------------------------------------------------------+
| -   Key to the success of this project was the timely availability,   |
|     commitment and response from senior executives.                   |
+-----------------------------------------------------------------------+
| -   The project established a common understanding and vision; gained |
|     senior management commitment and involvement in the process; and  |
|     demonstrated relevant and tangible benefits to the company's      |
|     management team.                                                  |
+-----------------------------------------------------------------------+
| -   The project revealed additional opportunities for improvement of  |
|     processes and systems, especially strategic planning and          |
|     budgeting, and early warning of impending risks.                  |
+-----------------------------------------------------------------------+

### Case 2: Global Marketer, Producer, and Distributor of Beverages

+-----------------------------------------------------------------------+
| **Business Challenge:**                                               |
+-----------------------------------------------------------------------+
| -   Desired an assessment of current Enterprise Risk Management (ERM) |
|     initiatives compared to leading practices                         |
+-----------------------------------------------------------------------+
| -   Existing ERM program defined as the "road to nowhere"; seeking    |
|     assistance to gain executive and operating unit buy-in            |
+-----------------------------------------------------------------------+
| -   Selection criteria dependent upon demonstrating both beverage and |
|     ERM understanding                                                 |
+-----------------------------------------------------------------------+
| **Solution:**                                                         |
+-----------------------------------------------------------------------+
| -   Thought leadership and experience providing ERM services          |
+-----------------------------------------------------------------------+
| -   Use of beverage industry knowledge and experience                 |
+-----------------------------------------------------------------------+
| -   Ability to indicate value-added risk services                     |
+-----------------------------------------------------------------------+
| -   Culture of communication and collaboration                        |
+-----------------------------------------------------------------------+
| -   Tools to identify risk, assess risk management capability, link   |
|     these to key value drivers                                        |
+-----------------------------------------------------------------------+
| **Results:**                                                          |
+-----------------------------------------------------------------------+
| -   Identified a process or methodology that ensures critical issues  |
|     are raised quickly to senior management and the Board of          |
|     Directors                                                         |
+-----------------------------------------------------------------------+
| -   Recommendations to improve the effectiveness of the Company's ERM |
|     initiative including organizational structure, leadership         |
|     involvement, and project management                               |
+-----------------------------------------------------------------------+
| -   Develop a roadmap that will successfully lead the company from    |
|     its current state to the desired future state                     |
+-----------------------------------------------------------------------+
| -   The cycle of risk identification, assessment, response, and       |
|     monitoring is embedded in the organization's standard strategic   |
|     planning and budgeting processes                                  |
+-----------------------------------------------------------------------+
| **Organizational Learning:**                                          |
+-----------------------------------------------------------------------+
| -   Listening to the needs of executives and following up with them   |
|     in a consistent, thoughtful manner                                |
+-----------------------------------------------------------------------+
| -   Providing the best resources available and working with executive |
|     teams throughout the entire process, end-to-end                   |
+-----------------------------------------------------------------------+
| -   Providing a well-thought out roadmap that will detail the steps   |
|     needed to implement a sustainable, ongoing ERM program            |
+-----------------------------------------------------------------------+

### Case 3: State Retirement Agency

+-----------------------------------------------------------------------+
| **Business Challenge:**                                               |
+-----------------------------------------------------------------------+
| -   Assist the Board of Trustees in ensuring that management has an   |
|     effective program to identify, assess, prioritize, and respond to |
|     risk                                                              |
+-----------------------------------------------------------------------+
| -   Identify significant business risks associated with the Agency's  |
|     strategic objectives and operations                               |
+-----------------------------------------------------------------------+
| -   Provide input to a formal risk response approach and action plans |
+-----------------------------------------------------------------------+
| -   Understand the Agency's current state and future vision for its   |
|     overall risk management approach                                  |
+-----------------------------------------------------------------------+
| -   Provide the baseline for a sustainable risk assessment and risk   |
|     management methodology in which the Agency's management will be   |
|     trained                                                           |
+-----------------------------------------------------------------------+
| **Solution:**                                                         |
+-----------------------------------------------------------------------+
| -   Develop a specific risk framework that includes risk categories   |
|     and definitions to promote a common language of risk and a better |
|     basis for on-going risk reporting and monitoring                  |
+-----------------------------------------------------------------------+
| -   Produce a high-level, yet focused risk assessment process that    |
|     enables the company to identify its mission critical risks and    |
|     priorities for mitigation and assurance of its Agency Risk        |
|     Profile.                                                          |
+-----------------------------------------------------------------------+
| -   Facilitate an assessment of risk management capability to         |
|     facilitate identification of improvement opportunities across its |
|     risk management and governance programs and to provide high level |
|     recommendations for sustainable capability.                       |
+-----------------------------------------------------------------------+
| -   Provide a repeatable and practical risk assessment process that   |
|     includes training and basic tools                                 |
+-----------------------------------------------------------------------+
| -   Provide recommendations regarding organizational options to help  |
|     sustain an Agency-wide Risk Management process                    |
+-----------------------------------------------------------------------+
| **Results:**                                                          |
+-----------------------------------------------------------------------+
| -   Developed and applied an Agency-wide risk assessment process      |
+-----------------------------------------------------------------------+
| -   Developed and delivered orientation and training to 45 managers   |
|     and executives                                                    |
+-----------------------------------------------------------------------+
| -   Developed a framework and recommendations for sustaining an       |
|     improved risk management process at the agency                    |
+-----------------------------------------------------------------------+
| -   Prepared and delivered assessment results to the Board of         |
|     Trustees and helped gain acceptance of management's response      |
|     plans                                                             |
+-----------------------------------------------------------------------+
| -   The Agency has released an RFP for the immediate replacement of   |
|     its retirement information system.                                |
+-----------------------------------------------------------------------+
| **Organizational Learning:**                                          |
+-----------------------------------------------------------------------+
| -   Quality and flexibility in approach required to adapt methodology |
|     to unique needs for a public sector client                        |
+-----------------------------------------------------------------------+
| -   Focus on detailed project planning and team training              |
+-----------------------------------------------------------------------+
| -   Use of assessment to position corrective action and continuous    |
|     process and system improvement at the Agency.                     |
+-----------------------------------------------------------------------+

-   

## 3.7 NIH specific Information

In addition to the four external sources of information, interviews with
NIH directors and review of NIH documents rounded out data collection
and analysis. In this section are included high level themes from
interviews and document reviews. Also developed are a set of baseline
risks mentioned in interviews and evident in documents.

+-----------------------------------------------------------------------+
| **Key Observations from this Study**                                  |
+=======================================================================+
| -   NIH has an executive office structure composed of several offices |
|     of the director that manage extramural and intramural research,   |
|     technology, special and emerging issues, administrative and       |
|     financial programs. The Office of the Director is supported by    |
|     various advisory groups. The administrative, financial and        |
|     support offices, as well as certain emerging issues offices and   |
|     centers report to the Deputy Director, who in turn reports to the |
|     NIH Director. Directors lead over 27 institutes and centers and   |
|     report to the NIH Director. Many institutes and centers have      |
|     their own extramural, intramural, financial, performance,         |
|     technology and other administrative offices, divisions and        |
|     branches to support specific institute and center mission         |
|     requirements.                                                     |
+-----------------------------------------------------------------------+
| -   NIH employs four types of Advisory Committees:                    |
|                                                                       |
|     -   Integrated/Initial Review Groups (IRGs) and Special Emphasis  |
|         Panels (SEPs)                                                 |
|                                                                       |
|     -   Boards of Scientific Counselors (BSCs)                        |
|                                                                       |
|     -   Program Advisory Committees (PACs)                            |
|                                                                       |
|     -   National Advisory Councils and Boards (NACs)                  |
+-----------------------------------------------------------------------+
| -   Committee management at NIH resides at the NIH Office of the      |
|     Director (OD) level in the Office of Federal Advisory Committee   |
|     Policy and the Institute or Center (IC) level with the Committee  |
|     Management Office (CMO). OFACP is responsible for the oversight   |
|     of all NIH Federal advisory committees under the auspices of the  |
|     FACA.                                                             |
+-----------------------------------------------------------------------+
| -   At the IC level, each IC has a CMO or uses the resources of a     |
|     service center, to support the committee management function      |
|     within the Institute or Center.                                   |
+-----------------------------------------------------------------------+
| -   A program management office (PMO) has been established to set the |
|     direction of primary ERM activities and to promote a coordinated  |
|     and leveraged approach to ERM across NIH.                         |
+-----------------------------------------------------------------------+
| -   Senior NIH Management is beginning to be effectively engaged in   |
|     the ERM process, serving in some aspects of a risk oversight      |
|     council.                                                          |
+-----------------------------------------------------------------------+
| -   Lists of risks may or may not reflect interdependences or be      |
|     mapped to processes and systems, be driven by data where          |
|     feasible, or be linked to standard organization and               |
|     process/system representations of NIH.                            |
|                                                                       |
| -   Controls appear to be appropriately tested across key processes,  |
|     systems and functions throughout NIH.                             |
+-----------------------------------------------------------------------+
| -   Deficient conditions appear to be identified as a result of       |
|     assurance activities. Conditions appear to be promptly            |
|     investigated and corrective actions seem to be formulated.        |
+-----------------------------------------------------------------------+
| -   NIH currently uses management control based areas of risk.        |
+-----------------------------------------------------------------------+
| -   In some offices, current lists of risks are generated by          |
|     reference to typical events, recent experiences within and        |
|     outside of NIH.                                                   |
+-----------------------------------------------------------------------+
| -   Various NIH offices and other organizational units employ         |
|     checklists to self-assess and self-report risks.                  |
+-----------------------------------------------------------------------+
| -   Some units use severity and probability as key dimensions for     |
|     risk assessment                                                   |
+-----------------------------------------------------------------------+
| -   NIH has various detailed programs of improvement resulting from   |
|     activities that may include for example: internal reviews,        |
|     internal or external assessments, user feedback, complaints and   |
|     other issues.                                                     |
+-----------------------------------------------------------------------+
| -   NIH does not appear to have an NIH-wide approach for aggregating, |
|     reporting, or responding to risks or generating risk response     |
|     alternatives.                                                     |
+-----------------------------------------------------------------------+
| -   Many of offices report ad hoc approaches to generating risk       |
|     responses, and some highly specialized approaches to mitigating   |
|     risks (e.g., ORF, Acquisitions, Technology).                      |
+-----------------------------------------------------------------------+
| -   NIH is in the midst of changing its risk approach and harmonizing |
|     the generation of risk response.                                  |
+-----------------------------------------------------------------------+

NIH managers have previously identified 50 risk areas. On May 6, 2006,
the following examples of risk areas were noted as Medium (M) or High
(H):

> • NIH-Wide Risk Management Assessment (M)
>
> • Technology Transfer (H)
>
> • Gift Funds (M)
>
> • Fellowship Stipend Payments (H)
>
> • Calling Cards (H)
>
> • Convenience Checks (H)
>
> • Assessing and Documenting Internal Controls over Financial Reporting
> (M)
>
> • Accounts Payable End-to-End Process Risk Assessment (M)

These are management control areas and response to known risks. Each has
a risk owner and status associated with a risk response plan.

In addition to these areas and based on interviews with several
administrative offices, the following areas were raised as high enough
level of risk to be mentioned in interviewees' "worst nightmares":

+-------------------------------------+--------------------------------+
| -   Grant/Contractor Indirect cost  | -   Outdated manual chapters   |
|     rates                           |     (50%) (e.g., Grants        |
|                                     |     Rescinded #54700 7/8/1985) |
| -   Grant Management activities     |                                |
+-------------------------------------+--------------------------------+
| -   Backlog in facility maintenance | -   Lack of data / business    |
|     (\$400M)                        |     measures                   |
+-------------------------------------+--------------------------------+
| -   Bringing NBS Online in 2007     | -   Peer Review                |
+-------------------------------------+--------------------------------+
| -   Recruitment, Retention,         | -   MTAs, CRADAs and           |
|     Retirement                      |     intellectual property      |
+-------------------------------------+--------------------------------+
| -   Ethical Conduct / Conflicts of  | -   A76 key person succession  |
|     Interest                        |                                |
+-------------------------------------+--------------------------------+
| -   Human Subjects and Animal Care  | -   IT governance              |
+-------------------------------------+--------------------------------+
| -   Fraud, Theft and Loss of NIH    | -   Regulatory requirements    |
|     property and supplies           |     for physical security      |
+-------------------------------------+--------------------------------+
| -   Clinical Trial practices        |                                |
+-------------------------------------+--------------------------------+

Other areas, process and system segments, and specific risks may have
been raised, but the purpose of this study is to only provide a high
level set of immediately reported issues to make concrete the
development of the methodology and test that these risks could be
surfaced using the proposed risk assessment methodology. A report of a
high level media scan regarding certain of these risks appears in
Appendix C.

#  

# 4. Summary of Recommendations

This section reports a summary of recommendations. Recommendations are
based on a selection of practices for organizations with complex
missions and structure, are consistent with guidance and standards, and
reflect recent organizational experience with the development of risk
assessment and management methodologies.

## 4.1 Recommendations Based on 8 Dimensions of Risk Intelligence

+-----------------------------------------------------------------------+
| **Key Recommendations from this Study**                               |
+=======================================================================+
| **Governance and Risk Oversight Recommendations**                     |
|                                                                       |
| 1.  The Oversight Group should annually review and approve NIH risk   |
|     management policy. This policy should clearly define risk, NIH's  |
|     philosophy about risk, its risk tolerance, its appetite for       |
|     various types of risks and the relationship between mission       |
|     critical risks and stakeholder value drivers.                     |
|                                                                       |
| 2.  The Oversight Group should approve a risk management policy that  |
|     outlines the mechanisms to delegate authorities and to elevate    |
|     issues and conflicts.                                             |
|                                                                       |
| 3.  The Oversight Group should review NIH's budgetary and other       |
|     financial objectives to ensure compatibility with the level of    |
|     risk embedded in NIH's annual plan.                               |
+-----------------------------------------------------------------------+
| **Risk Identification** **Recommendations**                           |
|                                                                       |
| 1.  Executive management should identify the key performance drivers  |
|     of NIH and the associated risks and scenarios by process, system, |
|     budget area, organization, mission objective, and other relevant  |
|     segments such as applicable laws, regulations and policies.       |
|                                                                       |
| 2.  Taxonomy of risks relevant to NIH's activities, industry          |
|     participation, and mission objectives should be updated regularly |
|     based on the volatility of NIH's objectives, budget, personnel    |
|     and its operating, regulatory and stakeholder environment.        |
|                                                                       |
| 3.  The risk taxonomy should be revised based on emerging and         |
|     experienced risks on at least an annual basis, ratified by a the  |
|     Director, NIH and the NIH Advisory Committee.                     |
+-----------------------------------------------------------------------+
| **Risk Assessment and Measurement Recommendations**                   |
|                                                                       |
| 1.  NIH-wide risks should be assessed on the basis of gross impact    |
|     (ie., gross risk, inherent risk, before mitigation and controls), |
|     net impact (ie, net risk, residual risk, vulnerability- after     |
|     mitigation and controls).                                         |
|                                                                       |
| 2.  Assessments can be a combination of qualitative and quantitative  |
|     and should be facilitated externally (for objectivity &           |
|     independence) not simply self-administered and self-reported.     |
|                                                                       |
| 3.  Wherever possible NIH should drive assessments with process,      |
|     system, budgetary, and external factor data, as available from    |
|     independent and controlled sources.                               |
|                                                                       |
| 4.  Assessment scales should be founded on performance and risk       |
|     indicators, and where not available, based on consistently        |
|     constructed data-equivalent word descriptions of events.          |
|                                                                       |
| 5.  Gross risk (before risk response and controls) should be assessed |
|     at least in relation to quantitative factors such as process      |
|     contribution, organization size, workflow volume (e.g., grants,   |
|     facilities) and cost of prior risk experience (direct hits and    |
|     near misses), as well as qualitative factors such as speed of     |
|     onset of the threat, as well as impacts on key stakeholders,      |
|     reputation, legal/regulatory, environment, health and safety and  |
|     speed of onset.                                                   |
|                                                                       |
| 6.  Net risk or vulnerability (after controls) should be assessed in  |
|     relation to such factors as Control Effectiveness (especially     |
|     relative to People, Process and Systems) Speed of Response        |
|     (Detection, Response, Recovery), Complexity or volatility of      |
|     activities, and Geographical dispersion, Response to Prior Risk   |
|     Experience, Rate of Internal Change and External Conditions.      |
|                                                                       |
| 7.  Net risk should be assessed relative to risk appetite and further |
|     mitigated as needed                                               |
|                                                                       |
| 8.  Mitigated value should be calculated, i.e., the difference        |
|     between gross and net risk.                                       |
|                                                                       |
| 9.  Internal and external subject matter specialists should be        |
|     involved as appropriate in the assessment of risk.                |
|                                                                       |
| 10. NIH's exposure to extreme upsides and downsides should be         |
|     assessed on a regular basis relative to intolerable and tolerable |
|     risk categories, for tolerable risks according to NIH risk        |
|     appetite.                                                         |
|                                                                       |
| 11. Probabilistic analyses should only be used when appropriate,      |
|     e.g., when there is operative a law of large numbers, and cause   |
|     and effect relationships are known, as well as other assumptions  |
|     required for meaningful use of probabilistic analysis.            |
|                                                                       |
| 12. Statistical risk measures should always be supplemented with      |
|     stress-testing and scenario analysis especially for extreme gross |
|     and net risks.                                                    |
|                                                                       |
| 13. Risk measures should be adapted to the types of risks taken       |
|     especially in relation to the mission specific objectives of      |
|     offices, centers and institutes and their use of otherwise common |
|     processes and systems.                                            |
|                                                                       |
| 14. Root Cause Analysis should be performed when significant risks do |
|     occur.                                                            |
|                                                                       |
| 15. Estimates of gross and net risk should be revised based on        |
|     experience                                                        |
|                                                                       |
| 16. A systematic independent verification of the risk assessment      |
|     process and results should be used to validate relevance and      |
|     efficiency.                                                       |
+-----------------------------------------------------------------------+
| **Response Alternatives and Response Plan Recommendations**           |
|                                                                       |
| 1.  Alternative risk responses should be formulated and evaluated for |
|     alignment with the NIH\'s risk tolerance and risk appetite.       |
|                                                                       |
| 2.  Risk responses should be organized in a hierarchy of response,    |
|     i.e., from lowest to highest strength of response, cost, and      |
|     other factors consistent with NIH policy, risk appetite and       |
|     culture.                                                          |
|                                                                       |
| 3.  NIH management should determine priorities taking into account    |
|     such factors as speed of risk onset, urgency, cost of mitigation  |
|     compared to expected benefit, degree of difficulty and time       |
|     required to implement.                                            |
|                                                                       |
| 4.  Responses should be integrated to provide effective and timely    |
|     NIH-wide preparation, response and recovery.                      |
+-----------------------------------------------------------------------+
| **Control Activities, Assurance** **and Testing Recommendations**     |
|                                                                       |
| 1.  Management should identify appropriate control activities to      |
|     ensure that its risk responses are carried out properly and in a  |
|     timely manner for all mission-critical risks at NIH.              |
|                                                                       |
| 2.  Mandatory disclosure requirements should be met in a timely       |
|     fashion.                                                          |
|                                                                       |
| 3.  General and application controls, including preventive,           |
|     detective, manual, computer and management controls should be     |
|     clearly defined in policies and procedures for each               |
|     mission-critical risk, relative to each process/system segment,   |
|     organizational unit, and budgetary authority.                     |
|                                                                       |
| 4.  Control activities should be matched to the speed of risk onset,  |
|     not the speed of response.                                        |
+-----------------------------------------------------------------------+
| **Risk Intelligence, Communication** **and Training Recommendations** |
|                                                                       |
| 1.  Risk intelligence should be gathered internally and externally.   |
|                                                                       |
| 2.  Risk intelligence should be evaluated for systemic bias, e.g.,    |
|     credibility of the source (including trustworthiness and          |
|     competence).                                                      |
|                                                                       |
| 3.  Risk intelligence should be available on a timely basis relative  |
|     to speed of risk onset.                                           |
|                                                                       |
| 4.  Risk intelligence should be integrated within and across core     |
|     decision-making processes and risk specializations.               |
|                                                                       |
| 5.  Performance Metrics / Dashboards / Scorecards should be           |
|     established for all mission-critical risks, especially as they    |
|     are segmented by processes and systems, relative to               |
|     accountabilities and incentives for improvement. Metrics may be   |
|     both qualitative and quantitative. Each scorecard element should  |
|     include NIH's risk appetite, actual current results, previous     |
|     results and target metric.                                        |
+-----------------------------------------------------------------------+
| **Monitoring and Escalation** **Recommendations**                     |
|                                                                       |
| 1.  NIH management, risk personnel and risk oversight groups should   |
|     monitor mission critical risks to review and evaluate performance |
|     compared to NIH objectives.                                       |
|                                                                       |
| 2.  NIH management should monitor mission-critical risks to review    |
|     and evaluate adherence to applicable laws, regulations, policies  |
|     and procedures.                                                   |
|                                                                       |
| 3.  NIH management should monitor mission-critical risks to review    |
|     and evaluate whether or not substantial progress is being made in |
|     managing risk exposures so that they are within NIH's appetite    |
|     for such exposures, if tolerable.                                 |
+-----------------------------------------------------------------------+
| **Sustainability and Continuous Improvement** **Recommendations**     |
|                                                                       |
| 1.  Lessons learned should be identified and communicated to          |
|     appropriate personnel on a timely basis.                          |
|                                                                       |
| 2.  Failures to correctly identify and assess risks should be         |
|     investigated and immediately remediate.                           |
|                                                                       |
| 3.  Risk related policies and procedures should be reviewed and       |
|     updated on a timely basis.                                        |
|                                                                       |
| 4.  Management should maintain a risk intelligence knowledge base     |
|     embedded in core NIH management processes.                        |
+-----------------------------------------------------------------------+

##  

## 4.2 Proposed Enterprise Risk Assessment Process

Based on observations from guidance and recent case experience, here is
a 5 step process that incorporates insights from decision science,
economics, psychology and sociology.

##   {#section-5 .unnumbered}

## 4.3 Three Options

Using the 8 dimensions of the risk intelligent organization, the team
has formulated three options for performing the NIH-wide risk scan. Each
option follows the process outlined above -- Proposed Enterprise Risk
Assessment Process, but may differ considerably in terms of tools and
data requirements. As one evaluates and compares the three options,
he/she will notice increases from A through C in data collection
sophistication, level of effort, and rigor of analyses.

+-------------------------+----------------------+--------------------+
| ### Option A: Initiate  | ### Option B: Ini    | ### Option C: De   |
| Facilitated Assessment: | tiate Data-Driven Fa | velop Comprehensiv |
|                         | cilitated Assessment | e Data-Driven Asse |
|                         |                      | ssment Environment |
+-------------------------+----------------------+--------------------+
| -   Perform NIH-wide    | -   **Option A plus  | -   **Option B     |
|     risk assessment at  |     the following    |     plus the       |
|     least annually      |     items:**         |     following      |
|                         |                      |     items:**       |
| -   Base assessment on  | -   Assign           |                    |
|     key risks           |     quantitative     | -   Use data from  |
|     identified by OIR,  |     operational and  |     NBS[^34] and   |
|     OER, and other      |     key risk         |     other sources  |
|     offices, centers    |     indicator        |     collated by    |
|     and institutes      |     measurements, as |     the NIH-wide   |
|                         |     possible, to key |     risk           |
| -   Replace existing    |     risks and their  |     assessment     |
|     assessment          |     mitigations      |     team to        |
|     checklists with key |     through risk     |     populate a     |
|     risks assessment    |     responses        |     risk scorecard |
|     survey              |                      |                    |
|                         | -   Use the          | -   Based on       |
| -   Replace Yes/No      |     assignment of    |     re-arranged    |
|     answer formats with |     quantitative     |     triggers       |
|     5-point Likert[^33] |     measurements to  |     operating      |
|     scales based on     |     populate a       |     within the     |
|     qualitative data    |     rudimentary risk |     scorecard,     |
|     gathered from OIR,  |     scorecard for    |     including the  |
|     OER, and other      |     tracking,        |     results of the |
|     offices, centers    |     incentive,       |     last           |
|     and institutes      |     compliance,      |     assessment and |
|                         |     accountability   |     additional     |
| -   Assign executive    |     and planning     |     data gathering |
|     risk owners and     |     purposes         |     work by the    |
|     subject matter      |                      |     NIH-wide risk  |
|     specialists to each | -   Populate         |     assessment     |
|     key risk and        |     assessment       |     team, develop  |
|     require annual      |     rating scales    |     risk           |
|     assessment of key   |     (Likert-style)   |     assertions and |
|     risks as input to   |     with operational |     challenge      |
|     Work Group          |     data, especially |     responses for  |
|     considerations      |     thresholds for   |     consideration  |
|                         |     risk appetite,   |     by executive   |
| -   Track on a NIH-wide |     applicable to    |     risk owners    |
|     basis the           |     assessor         |                    |
|     identification of   |     environments     | -   Build internal |
|     plans to manage key |                      |     controls into  |
|     risks, including    | -   Require          |     all major      |
|     the further         |     executive risk   |     systems to     |
|     consideration of    |     owners to trend  |     automatically  |
|     risks not on the    |     operational and  |     report red     |
|     key risk list, and  |     key risk         |     flags that     |
|     the progress to     |     indicator        |     require        |
|     date of plans       |     measurements as  |     management     |
|                         |     an input to      |     attention.     |
| -   Enable the tracking |     their assessment |                    |
|     with at least a     |     of key risks     | -   Example: The   |
|     qualitative         |                      |     Office of      |
|     measurement based   | -   Link risk        |     Research       |
|     risk scorecard      |     response plans   |     Facilities.    |
|                         |     by executive     |                    |
| -   Align tracking of   |     risk owners to   | ###                |
|     risk improvement    |     quantitative     |                    |
|     plans to budgets to |     operational      |                    |
|     accomplish risk     |     goals and        |                    |
|     responses,          |     improvement in   |                    |
|     including funding   |     key risk         |                    |
|     through avoided     |     indicators       |                    |
|     cost and process    |     assigned to each |                    |
|     savings resulting   |     process/system   |                    |
|     from increased risk |     relative to      |                    |
|     management          |     various          |                    |
|     effectiveness and   |     organizational   |                    |
|     efficiency          |     units            |                    |
|                         |                      |                    |
| -   To provide          | -   Track progress   |                    |
|     sufficient          |     of corrective    |                    |
|     challenge to        |     action plans     |                    |
|     executive risk      |     using            |                    |
|     owners, subject     |     quantitative     |                    |
|     matter specialists, |     operational      |                    |
|     both internal to    |     measurements     |                    |
|     NIH and external,   |                      |                    |
|     can be effectively  | -   Example: The     |                    |
|     deployed provide    |     processes and    |                    |
|     risk information    |     systems in the   |                    |
|     and facilitation    |     Office of        |                    |
|                         |     Extramural       |                    |
| -   Example: The        |     Research and in  |                    |
|     processes and       |     extramural       |                    |
|     systems in the      |     research         |                    |
|     Office of           |     processes and    |                    |
|     Intramural Research |     systems in       |                    |
|     and in Intramural   |     various          |                    |
|     research processes  |     institutes and   |                    |
|     and systems in      |     centers.         |                    |
|     various institutes  |                      |                    |
|     and centers.        |                      |                    |
+-------------------------+----------------------+--------------------+

***4.4 Proposed Governance Structure***

The proposed NIH methodology, as described in the section on Procedures,
requires the use of several roles and entities summarized and depicted
below.

-   **Advisory Committee \--** A person or persons accountable by the
    organization for prioritizing risks, assigning executive risk owners
    and assigning responsibility for actions taken to correct an
    unacceptable level of risk.

-   **Executive Team** -- a group that supports the AC, provides subject
    matter assistance, and may assist in the direction of specific risk
    assessment activities.

-   **Risk Area Manager** -- A person assigned by the executive sponsor
    accountable for the assessment of a specific risk and any actions
    taken to correct an unacceptable level of risk.

-   **Subject Matter Specialist** -- A person assigned by an executive
    risk owner able to provide inherent and residual and capability
    assessment input to an organization's specific risk during the Risk
    Prioritization phase of the planning process.

-   ![](media/image35.wmf)**Assessment Team --** The team that
    facilitates the NIH-wide risk assessment and management process.
    Elements of this team may include OMA, RMOs and others designated by
    the Director of NIH.

NIH will require a sound governance structure to effectively administer,
monitor, and evaluate risks. At a high level, the key action steps for
designing and implementing an effective governance structure are:

1.  Outline key needs and responsibilities of the various entities

2.  Develop governance charters for each entity

3.  Identify key individuals to provide adequate perspectives and
    representation

4.  Prepare meeting schedules, operating procedures, and reporting
    mechanisms

5.  Begin implementing governance structure and conducting meetings

## 4.5 Study Questions and Summary Answers

During this study, the following questions are answered:

> *1. What is the best methodology for the NIH to use to evaluate risk
> for its scientific, administrative, and financial programs for the
> NIH-wide risk assessment in Phase 2, and in the future?*

-   Leading practice methodology for NIH exists in form of maturing
    standards and guidance, and examples of implementations at large,
    complex organizations

-   The methodology includes governance, risk identification, risk
    assessment and measurement, risk response alternatives and plans,
    monitoring and escalation, control activities, assurance and
    testing, risk intelligence, communications and training, and
    sustainability and continuous improvement

-   These methodology components are listed using the 8 dimensions of
    the risk intelligent organization, are verified though a preliminary
    survey of over 120 organizations and case example

> *2. What is the most economical, efficient and effective way to
> collect the data needed to evaluate these high risk areas without
> imposing an excessive burden on program staff or NIH administrators?*

-   Based on leading practice methodology components a 5 step process is
    proposed

-   The process is a facilitated assessment and can be implemented using
    3 options, also presented, in order of increasing data, information
    , and intelligence-driven sophistication

-   Using the 3 options the 5 step process can be economically,
    efficiently, and effectively deployed in a scalable fashion across
    all of NIH and within individual institutes, centers and offices

-   Only by embedding this process into the regularly occurring NIH
    strategy, operations, scientific, financial, budgeting, and
    administrative process and system

> *3.. What is the baseline status of the existing risk areas previously
> identified by NIH managers?*

-   This study, through media scans, reference to subject matter
    specialists, review of NIH existing documentation of risk areas and
    other documents, and interviews with NIH executives has produced a
    list of baseline risk examples for review by NIH

-   Among other areas, processes and systems associated with grants,
    human resources, emergency preparedness, strategic planning and
    budgeting appear to be mentioned by NIH executives as very high
    priority risk areas for consideration

## 

# 5. Summary of References Consulted

Aamodt, A. and E. Plaza (1994) \"Case-Based Reasoning: Foundational
Issues, Methodological Variations, and System Approaches,\" *Artificial
Intelligence Communications* 7: 1, 39-52.

Carr, M.J., Konda, S.L., Monarch, I.A., Ulrich, F.C. and Walker, C.F.
(1993) *Taxonomy-Based Risk Identification, SEI Technical Report
SEI-93-TR-006*, Pittsburgh, PA: Software Engineering Institute.

Camerer, C. and M. Weber (1992), Recent developments in modelling
preferences: Uncertainty and ambiguity. In: Journal of Risk and
Uncertainty, 5, 325-370.

Camerer, Colin F. (1997), Progress in Behavioral Game Theory. In:
Journal of Economic Perspectives 11, 4, 167-188.

Davis, Douglas D. and Holt, Charles A. (1993), Experimental Economics.
Princeton, NJ: Princeton University Press.

Fischhoff, Baruch, A. Bostrom, M.J. Quadrel (2000), Risk Perception and
Communication. In: Connolly, Terry, H.R., Arkes, K. R. Hammond (eds.),
Judgment and Decision Making. An Interdisciplinary Reader, Cambridge:
Cambridge University Press, 479-499 (orig. 1993, in: Annual Review of
Public Health, 14, 183-203).

Frankfort-Nachmias, Chava, and David Nachmias (2000), Research Methods
in the Social Sciences 6^th^ Ed., St. Martins.

Hammond, Keeney, Raiffa (2006) The Hidden Traps in Decision Making,
Harvard Business Review.(January), 118-126.

Keeney, R.L. and Raiffa, H. (1976) *Decision with Multiple Objectives:
Preferences and Value Tradeoffs*, New York: John Wiley & Sons.

Kontio, J. (1994) Software Engineering Risk Management: A Technology
Review Report. PI_4.1, Helsinki, Finland: Nokia Research Center.

Lash, Scott (2000) Risk Culture. In: Adam, Barbara; Beck, Ulrich; Van
Loon, Joost (eds.): The Risk Society and Beyond. Critical Issues for
Social Theory. London, Thousand Oaks, New Delhi: Sage, 47-62.

Leonard, Robert J. (1995), From Parlor Games to Social Science: von
Neumann, Morgenstern, and the Creation of Game Theory. In: Journal of
Economic Literature XXXIII, 730-761.

Loewenstein, G. and Lerner, J. (2003), The role of affect in
decision-making, in R.Davidson,K.Scherr and H. Goldsmith (eds) Handbook
of Affective Studies, Oxford University Press.

Loewenstein, G., Weber, E., Hsee, C. and Welch, N. (2001), Risks as
feelings, Psychological Bulletin, 127, 2, 267-86.

Mayo,E. (1933) *The human problems of an industrial civilization* (New
York: MacMillan).

Miles, M.B. and A.M. Huberman (1994) Qualitative Data Analysis 2^nd^
ed., Thousand Oaks, CA: Sage.

Niven, P. (2003) Balanced Scorecard for Government and Nonprofit
Agencies, Hoboken, NJ: John Wiley and Sons.

Nutt, Paul (2002), Why Decisions Fail, Berrett-Koehler.

OCTAVE: Operationally Critical Threat, Asset, and Vulnerability
Evaluation. 2001, Carnegie Mellon University.

Plott, Charles R. (1996), Rational Individual Behavior in Markets and
Social Choice Processes: The Discovered Preference Hypothesis. in Arrow,
K. , E. Colombatto, M. Perleman, C. Schmidt (eds.), Rational Foundations
of Economic Behavior, London: Macmillan and New York: St. Martins,
225-50.

Risk Management: Australian Standards/New Zealand Standards 4360:2004.

Russo, J. Edward and Paul J. H. Schoemaker (1989) Decision Traps,
Doubleday.

Starmer, Chris (2000), Development in Non-Expected Utility Theory: The
Hunt for a Descriptive Theory of Choice under Risk. In: Journal of
Economic Literature, XXXVIII, 332-382.

The Green Book: Appraisal and Evaluation in Central Government, January
2003, Published with the permission of HM Treasury on behalf of the
Controller of Her Majesty's Stationery Office.

The Orange Book: Management of Risk - Principles and Concepts, October
2004, Published with the permission of HM Treasury on behalf of the
Controller of Her Majesty's Stationery Office.

Trochim, W.M.K. (2001) The Research Methods Knowledge Base 2^nd^ Ed.,
Cincinnati, OH: Atomic Dog Publishing.

Tversky, A. and D. Kahneman (1981), The framing of decisions and the
psychology of choice. In: Science, 211, 453 - 458.

Tversky, A. and D. Kahneman (1992), Advances in prospect theory:
Cumulative representation of uncertainty. In: Journal of Risk and
Uncertainty, 5, 297-323.

Tversky, A. and D. Kahneman (1974), Judgment Under Uncertainty:
Heuristics and Biases. In: Science 185, 1124-31.

Tversky, A. and D. Kahneman (1981), The framing of decisions and the
psychology of choice. Science 211, 453-458.

Wynne, Brian (1996), May the sheep safely graze? A reflexive view of the
expert-lay knowledge divide. In: Lash, S., B. Szerszinski, and B. Wynne,
B. (eds.), Risk, Environment and Modernity: Towards a New Ecology.
London: Sage, 44-83.

Vandenbroucke JP (2001) In defense of case reports and case series.
Annals of Internal Medicine 134(4):330-4.

# APPENDICES

A.  **Interviewees**

B.  **Documents Reviewed**

C.  **Summary High Level Media Scan**

D.  **Risk IQ**

## Appendix A: Interviewees (status as of August 5, 2006)

  ------------- ------------- ---------- ---------------- ----------------- ---------- ------------
  **Name**      **Title**     **NIH      **Branch**       **Division**      **Date**   **Status**
                              Office**                                                 

  Linda Adams   Management    OIR        OD                                 7/10/06    Complete
                Liaison                                                                

  Diane Dean    Director      OER        Office of Policy Division of       7/10/06    Complete
                                         for Extramural   Grants Compliance            
                                         Research         and Oversight                
                                         Administration                                

  Salley Rockey Deputy        OER        \-               \-                7/10/06    Complete
                Director                                                               

  Diane Frasier Director      OM         Office of        Director, Office  7/13/06    Complete
                                         Administration   of Acquisition               
                                                          Management and               
                                                          Policy                       

  Juanita       Acting        OM         Office of        Division of       7/14/06    Complete
  Mildenberg    Director                 Research         Property                     
                                         Facilities       Management                   

  Colleen       NIH Deputy    OM                                            7/21/06    Complete
  Barros        Director                                                               

  Ken Stith     Director      OM         Office of                          7/25/06    Complete
                                         Financial                                     
                                         Management                                    

  Christine     Acting        OM         Office of Human                    7/26/06    Complete
  Major         Director                 Resources                                     

  Jeff Weiner   Acting Deputy CIT        Office of the                      7/27/06    Complete
                Chief                    Deputy Chief                                  
                Information              Information                                   
                Officer                  Officer                                       

  Daniel Sands  Director      CIT        Office of the    Information       7/27/06    Complete
                                         Deputy Chief     Security                     
                                         Information      Awareness Office             
                                         Officer                                       

  Judy Blair    Senior        CIT        Office of the                      7/27/06    Complete
                Management               Deputy Chief                                  
                Analyst                  Information                                   
                                         Officer                                       

  Dona Lenkin                 CIT        OD                                 7/27/06    Complete

  Shirl Eller   Acting        OM         Office of                          7/28/06    Complete
                Director                 Research                                      
                                         Services /OD                                  

  Rich Southers Director      OM         Office of        Office of         7/28/06    Complete
                                         Research         Administrative               
                                         Services /OD     Management                   

  LaVerne       Associate     OD         Executive Office                   7/31/06    Complete
  Stringfield   Director for                                                           
                Management                                                             

  Andy Baldus   Acting        OM         Office of Budget                              Scheduled
                Associate                                                              
                Director                                                               
  ------------- ------------- ---------- ---------------- ----------------- ---------- ------------

## Appendix B: Documents Reviewed

  ------- -------------------------------------------------- --------------------- --------------------------------------------------------------------------------------------------------------------
  ** **   **Primary Document Name**                          **Subtitle**          **Additional information**

  1       COSO II (ERM)                                                             

  2       ANZ 4360:2004                                                             

  3       BearingPoint documentation                                                Management Control Study

  4       [[www.nih.gov]{.underline}](http://www.nih.gov/)                          

                                                                                   

  5       Directory of NIH Advisory Committees               NIH/OD/Office of      [ ]{.underline}
                                                             Federal Advisory      
                                                             Committee Policy,     
                                                             January 2006          

  6       Five Standards of Internal Control                 U.S. Government        
                                                             Accountability Office 

  7       Initial Survey of NIH Executive Officers and                              
          Office of Management Office Directors                                    

  8       Federal Managers\' Financial Integrity Act                                

  9       NIH FMFIA Annual Report                            Fiscal Year 2006,      
                                                             Internal Report, As   
                                                             of June 30, 2006      

  10      Effective Administrative Restructuring: Lessons    National Academy of    
          for the NIH Experience                             Public Administration 

  11      Gift Solicitation invites Ethics Debate            WashingtonPost.com,    
                                                             Monday, May 8, 2006   

  12      AHF Calls for End of NIH-Funded AIDS Vaccine Trial Yahoo Finance: Press   
                                                             Release, Monday May   
                                                             22, 4:03pm ET         

  13      NIH Statement Regarding House Hearing on Human     Email from ExecSec1,   
          Tissue Samples                                     June 13, 2006         

  14      Federal Employee Retirements-Expected Increase     U.S. General          Report to the Chairman, Subcommittee on Civil Service and Agency Organization, Committee on Government Reform, House
          Over the Next 5 years Illustrates need for         Accounting Office,    of Representatives
          Workforce Planning                                 April 2001            

  15      Anthrax: Federal Agencies have Taken Some Steps to U.S. Government       Testimony before the Subcommittee on National Security, Emerging Threats and International Relations, Committee on
          Validate Sampling Methods and to Develop a         Accountability        Government Reform, House of Representatives
          next-Generation Anthrax Vaccine                    Office, May 9, 2006   

  16      Letter: National Institute of Environmental health U.S. Government        
          Sciences-American Chemistry Council Donation       Accountability        
                                                             Office, September 30, 
                                                             2005                  

  17      Federal Research-NIH and EPA Ned to improve        U.S. Government       Report to Congressional Requesters
          Conflict of Interest Reviews for Research          Accountability        
          Arrangements with Private Sector Entities          Office, February 2005 

  18      Letter: Report Number A-02-05-02002                Department of Health  Review of Cost Sharing at Columbia University
                                                             and Human Services,   
                                                             Office of Inspector   
                                                             General, Office of    
                                                             Audit Services, March 
                                                             3, 2006               

  19      Report: Superfund Financial Activties at the       Department of Health   
          National Institute of Environmental health         and Human Services,   
          Sciences for Fiscal year 2005 (A-04-06-01023)      Office of Inspector   
                                                             General, May 2006     

  20      Report: Review of Subaward Costs Claimed by Roger  Department of Health   
          Williams Hospital on NIH Grant Number 5 P01        and Human Services,   
          HL56920-05                                         Office of Inspector   
                                                             General, February     
                                                             2006                  

  21      Report: Review of Subaward Costs Claimed by Yale   Department of Health   
          University on NIH Grant Number 5 P01 HL56920-05    and Human Services,   
                                                             Office of Inspector   
                                                             General, February     
                                                             2006                  

  22      Report: Review of Costs Claimed by Dartmouth       Department of Health   
          College for NIH Grant No. 5 P01 GM51630-06         and Human Services,   
                                                             Office of Inspector   
                                                             General, September    
                                                             2005                  

  23      Report: NIH Grants Management: Late Awards         Department of Health   
                                                             and Human Services,   
                                                             Office of Inspector   
                                                             General, May 2004     

  24      Report: NIH Grants Management: Late Closeouts      Department of Health   
                                                             and Human Services,   
                                                             Office of Inspector   
                                                             General, May 2004     

  25      Report: Outside Activities of Senior-Level NIH     Department of Health   
          Employees                                          and Human Services,   
                                                             Office of Inspector   
                                                             General, July 2005    

  26      Report: Recruiting Human Subjects: Pressures in    Department of Health   
          Industry-Sponsored Clinical Research               and Human Services,   
                                                             Office of Inspector   
                                                             General, June 2000    

  27      Testimony: FY 2007 Director\'s Budget Request      NIH, April 6, 2006    Fiscal Year 2007 Budget Request, House Subcommittee on Labor, HHS, Education Appropriations
          Statement                                                                

  28      Federal Register: Scientific Peer Review of        Vol. 69 No. 2, Monday  
          Research Grant Applications and Research and       January 5 2004, Rules 
          Development Contract Projects                      and Regulations       

  29      How Scientists are Selected for Study Section      NIH                    
          Service                                                                  

  30      Federal Register: Federal Advisory Committee       Vol. 66 No. 139,       
          management Final Rule                              Thursday, July 19,    
                                                             2001, Rules and       
                                                             Regulations           

  31      Management Control - Risk Management                                      

  32      Email: Rob Weymouth to Suzanne Servis              Re: Management         
                                                             Control Suggestions   
                                                             and Documents,        
                                                             Tuesday, July 27,     
                                                             2004                  

  33      Guidelines for Identifying Risk Areas              Draft-Revised          
                                                             12/15/05              

  34      Risk Management Executive Seminar (dry run)        OMA                    

  35      NIH Manual Chapters as of july 2006                OMA                    

  36      Standard for Internal Control in the federal       U.S. General           
          Government                                         Accounting Office,    
                                                             November 1999         

  37      Key Components of Effective Risk Management        OMA                    
          (laminated 8.5x11\")                                                     

  38      OMB Requests peer Review of Proposed Risk          Office of Management   
          Assessment Bulletin                                and Budget, January   
                                                             9, 2006               

  39      NIH Management Control Program: Goals, Risk        NIH Training Center,   
          Assessment, and Review Process                     9/93                  

  40      Presentation: National Institutes of Health - Risk Colleen Barros, May    
          Management Program, Briefing of Scientific         17, 2006              
          Directors                                                                

  41      Acquisition-Administrative Restructuring           New Approaches to     [[http://arac.nih.gov/acquisitions/acquisition.asp]{.underline}](http://arac.nih.gov/acquisitions/acquisition.asp)
                                                             Acquisitions          

  42      Presentation: Environmental Management System      Executive Officers\'   
                                                             Briefing, June 29,    
                                                             2005                  

  43      Grant Closeout White Paper                         14-Jul-06              

  44      Program/Project Risk Prevention Plan               Draft                  

  45      NIH Organization Handbook                          US Department of       
                                                             Health and Human      
                                                             Services, National    
                                                             Institutes of Health  
                                                             Manual 1123, April    
                                                             2006                  

  46      Risk Assessment Analysis                           1994                   

  47      Analysis of NIH Risk Assessments in OMA Files      Prepared by the        
                                                             National Academy of   
                                                             Public                
                                                             Administration,       
                                                             January 2006          

  48      OMB Circular No. A-123, Revised                    21-Dec-04              

  49      The NIH Risk Management Program                    Effectively Managing   
                                                             Program Risk.         
                                                             Management Concepts   
                                                             Training Guide        

  50      Risk Assessment Instruction (New Ongoing Process)  Draft 4/25/06          

  51      Management Control Areas and Subareas Analysis     Developed 062305 by    
                                                             DanielJ. Martin,      
                                                             Printed 6/27/2005     

  52      Clinical Trials Glossary                                                  

  53      NIH Pharms Out Profits                             6/12/2003             Keith Ashdown

  54      Tauzin, Greenwood Investigate Fairness of NIH      11/10/2003            James C. Greenwood
          Award Process                                                            

  55      Post-Doubling, NIH Could Face Bumpy Ride           12/2/2003             AAAS

  56      Senate Panel Explores NIH Response to Allegations  1/23/2004             Dave Moore, AAMC
          of Conflicts of Interest                                                 

  57      Harvard Agrees to Pay \$2.4-Million More to Settle 6/21/2004             Jeffrey Brainard
          Allegations of Overcharging the NIH                                      

  58      National Institutes of Health criticised for not   7/3/2004              Janice Hopkins Tanne
          preventing conflicts of interest                                         

  59      NIH Rejects Request to Override Abbott\'s Norvir   8/5/2004              Kaiser Daily
          Patent                                                                   

  60      NIH Chimp Cruelty                                  9/7/2004              IDA

  61      Fatally flawed legal analysis will not stand in    11/18/2004            Bob Witeck
          the way of NIH public access plan                                        

  62      Allegations Of Waste, Fraud and Abuse              1/4/2005              Sophocles

  63      Royalty payments to staff researchers cause new    1/22/2005             Janice Hopkins Tanne
          NIH troubles                                                             

  64      Flawed AIDS Drug Study Exposes NIH Misconduct      2/8/2005              Jonathan M. Fishbein

  65      New Rules Will Cost Dissidents at NIH              3/3/2005              David Willman

  66      NIH Women Describe Sex Harrassment                 4/10/2005             John Solomon

  67      New rules drive off NIH researchers                4/16/2005             Janice Hopkins Tanne

  68      Director of NIH Agrees To Loosen Ethics Rules      8/26/2005             Ceci Connolly

  69      Grassley Hails NIH Reinstatement of Expert         12/28/2005            Associated Press

  70      NIH Scientist Violated Ethics Rules, Federal Laws, 6/14/2006             Kaisernetwork.org
          Congressional Report Finds                                               

  71      Sex Discrimination Ruins Doctor\'s Career                                Jyotsna Sreenivasan

  72      Radiation Poisoning: NIH Case Ends With Mysteries                        Jocelyn Kaiser
          Unsolved                                                                 

  73      2007 OIG Work Plan, External Review Draft, Public                         
          Health Agencies                                                          

  74      2007 OIG Work Plan, External Review Draft,                                
          Departmentwide Audits and Other Departmentwide                           
          Studies                                                                  

  75      Mission Statement                                  National Cancer        
                                                             Institute (NCI)       

  76      Mission Statement                                  National Institute on  
                                                             Aging (NIA)           

  77      Mission Statement                                  National Eye           
                                                             Institute (NEI)       

  78      Mission Statement                                  National Heart, Lung,  
                                                             and Blood Institute   
                                                             (NHLBI)               

  79      Mission Statement                                  National Human Genome  
                                                             Research Institute    
                                                             (NHGRI)               

  80      Mission Statement                                  National Institute on  
                                                             Alcohol Abuse and     
                                                             Alcoholism (NIAAA)    

  81      Mission Statement                                  National Institute of  
                                                             Allergy and           
                                                             Infectious Diseases   
                                                             (NIAID)               

  82      Mission Statement                                  National Institute of  
                                                             Arthritis and         
                                                             Musculoskeletal and   
                                                             Skin Diseases (NIAMS) 

  83      Mission Statement                                  National Institute of  
                                                             Arthritis and         
                                                             Musculoskeletal and   
                                                             Skin Diseases (NIAMS) 

  84      Mission Statement                                  National Institute of  
                                                             Biomedical Imaging    
                                                             and Bioengineering    
                                                             (NIBIB)               

  85      Mission Statement                                  National Institute of  
                                                             Child Health and      
                                                             Human Development     
                                                             (NICHD)               

  86      Mission Statement                                  National Institute on  
                                                             Deafness and Other    
                                                             Communication         
                                                             Disorders (NIDCD)     

  87      Mission Statement                                  National Institute of  
                                                             Dental and            
                                                             Craniofacial Research 
                                                             (NIDCR)               

  88      Mission Statement                                  National Institute of  
                                                             Diabetes and          
                                                             Digestive and Kidney  
                                                             Diseases (NIDDK)      

  89      Mission Statement                                  National Institute on  
                                                             Drug Abuse (NIDA)     

  90      Mission Statement                                  National Institute of  
                                                             Environmental Health  
                                                             Sciences (NIEHS)      

  91      Mission Statement                                  National Institute of  
                                                             General Medical       
                                                             Sciences (NIGMS)      

  92      Mission Statement                                  National Institute of  
                                                             Mental Health (NIMH)  

  93      Mission Statement                                  National Institute of  
                                                             Neurological          
                                                             Disorders and Stroke  
                                                             (NINDS)               

  94      Mission Statement                                  National Institute of  
                                                             Nursing Research      
                                                             (NINR)                

  95      Mission Statement                                  National Library of    
                                                             Medicine (NLM)        

  96      Mission Statement                                  Center for             
                                                             Information           
                                                             Technology (CIT)      

  97      Mission Statement                                  Center for Scientific  
                                                             Review (CSR)          

  98      Mission Statement                                  John E. Fogarty        
                                                             International Center  
                                                             (FIC)                 

  99      Mission Statement                                  National Center for    
                                                             Complementary and     
                                                             Alternative Medicine  
                                                             (NCCAM)               

  100     Mission Statement                                  National Center on     
                                                             Minority Health and   
                                                             Health Disparities    
                                                             (NCMHD)               

  101     Mission Statement                                  National Center for    
                                                             Research Resources    
                                                             (NCRR)                

  102     Mission Statement                                  NIH Clinical Center    
                                                             (CC)                  

  103     Organizational Chart                               Office of the          
                                                             Director, NIH         

  104     Organizational Chart                               Office of Disease      
                                                             Prevention            

  105     Organizational Chart                               Office of Extramural   
                                                             Research              

  106     Organizational Chart                               Office of Intramural   
                                                             Research              

  107     Organizational Chart                               Office of Management   

  108     Organizational Chart                               Office of              
                                                             Administration        

  109     Organizational Chart                               Office of Financial    
                                                             Management            

  110     Organizational Chart                               Office of Human        
                                                             Resources             

  111     Organizational Chart                               Office of Research     
                                                             Services              

  112     Organizational Chart                               Office of Research     
                                                             Facilities            
                                                             Development and       
                                                             Operations            

  113     Organizational Chart                               Office of Science      
                                                             Policy                

  114     Organizational Chart                               Office of              
                                                             Communications and    
                                                             Public Liaison        

  115     Organizational Chart                               Office of Equal        
                                                             opportunity and       
                                                             Diversity Management  

  116     Organizational Chart                               Office of Program      
                                                             Coordination          

  117     Organizational Chart                               NIH Ethics Office      

  118     Organizational Chart                               Office of Portfolio    
                                                             Analysis and          
                                                             Strategic Initiatives 
  ------- -------------------------------------------------- --------------------- --------------------------------------------------------------------------------------------------------------------

ADD:

*Financial Management Systems: HHS Faces Many Challenges in Implementing
Its Unified Financial Management System*. 
[GAO-04-1089T](http://www.gao.gov/cgi-bin/getrpt?GAO-04-1089T) . 
Washington, D.C.: September 30, 2004.

## Appendix C: Summary High-Level Media Scan

  ---------- -------------- ---------------- ------------- ------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **NIH Risk                                                                   
  Events                                                                       
  (Last                                                                        
  three                                                                        
  years)**                                                                     

  **\#**     **Category**   **Title of the   **Date        **Author/           **WWW Link to the Article**
                            Article**        Article       Correspondent/      
                                             Published**   Source**            

  1          Legal          NIH Pharms Out   6/12/2003     Keith Ashdown       [[http://www.taxpayer.net/TCS/wastebasket/budget/2003-06-12taxol.htm]{.underline}](http://www.taxpayer.net/TCS/wastebasket/budget/2003-06-12taxol.htm)
                            Profits                                            

  2          Congress       Tauzin,          11/10/2003    James C. Greenwood  [[http://energycommerce.house.gov/108/Letters/11102003_1127.htm]{.underline}](http://energycommerce.house.gov/108/Letters/11102003_1127.htm)
                            Greenwood                                          
                            Investigate                                        
                            Fairness of NIH                                    
                            Award Process                                      

  3          Congress       Post-Doubling,   12/2/2003     AAAS                [[http://www.aaas.org/spp/cstc/news/articles2003/031202_nih.shtml]{.underline}](http://www.aaas.org/spp/cstc/news/articles2003/031202_nih.shtml)
                            NIH Could Face                                     
                            Bumpy Ride                                         

  4          Congress       Senate Panel     1/23/2004     Dave Moore, AAMC    [[http://www.aamc.org/advocacy/library/washhigh/2004/012304/\_2.htm]{.underline}](http://www.aamc.org/advocacy/library/washhigh/2004/012304/_2.htm)
                            Explores NIH                                       
                            Response to                                        
                            Allegations of                                     
                            Conflicts of                                       
                            Interest                                           

  5          Legal          Harvard Agrees   6/21/2004     Jeffrey Brainard    [[https://www-s.dmi.uiuc.edu/ars/about/caseharvard.html]{.underline}](https://www-s.dmi.uiuc.edu/ars/about/caseharvard.html)
                            to Pay                                             
                            \$2.4-Million                                      
                            More to Settle                                     
                            Allegations of                                     
                            Overcharging the                                   
                            NIH                                                

  6          Congress       National         7/3/2004      Janice Hopkins      [[http://bmj.bmjjournals.com/cgi/content/full/329/7456/10-b]{.underline}](http://bmj.bmjjournals.com/cgi/content/full/329/7456/10-b)
                            Institutes of                  Tanne               
                            Health                                             
                            criticised for                                     
                            not preventing                                     
                            conflicts of                                       
                            interest                                           

  7          Legal          NIH Rejects      8/5/2004      Kaiser Daily        [[http://www.hivandhepatitis.com/recent/pricing/080604_a.html]{.underline}](http://www.hivandhepatitis.com/recent/pricing/080604_a.html)
                            Request to                                         
                            Override                                           
                            Abbott\'s Norvir                                   
                            Patent                                             

  8          Activists      NIH Chimp        9/7/2004      IDA                 [[http://www.nihchimpcruelty.com/nih_chronology_17.html]{.underline}](http://www.nihchimpcruelty.com/nih_chronology_17.html)
                            Cruelty                                            

  9          Legal          Fatally flawed   11/18/2004    Bob Witeck          [[http://www.library.yale.edu/\~llicense/ListArchives/0412/msg00015.html]{.underline}](http://www.library.yale.edu/~llicense/ListArchives/0412/msg00015.html)
                            legal analysis                                     
                            will not stand                                     
                            in the way of                                      
                            NIH public                                         
                            access plan                                        

  10         Media          Allegations Of   1/4/2005      Sophocles           [[http://www.honestdoctor.org/allegations.html]{.underline}](http://www.honestdoctor.org/allegations.html)
                            Waste, Fraud and                                   
                            Abuse                                              

  11         Media          Royalty payments 1/22/2005     Janice Hopkins      [[http://bmj.bmjjournals.com/cgi/content/full/330/7484/162-a]{.underline}](http://bmj.bmjjournals.com/cgi/content/full/330/7484/162-a)
                            to staff                       Tanne               
                            researchers                                        
                            cause new NIH                                      
                            troubles                                           

  12         Whistle blower Flawed AIDS Drug 2/8/2005      Jonathan M.         [[http://www.foxnews.com/story/0,2933,146199,00.html]{.underline}](http://www.foxnews.com/story/0,2933,146199,00.html)
                            Study Exposes                  Fishbein            
                            NIH Misconduct                                     

  13         Media          New Rules Will   3/3/2005      David Willman       [[http://www.newsdesk.org/archives/000154.php]{.underline}](http://www.newsdesk.org/archives/000154.php)
                            Cost Dissidents                                    
                            at NIH                                             

  14         Legal          NIH Women        4/10/2005     John Solomon        [[http://www.sfgate.com/cgi-bin/article.cgi?f=/n/a/2005/04/10/national/w101247D57.DTL]{.underline}](http://www.sfgate.com/cgi-bin/article.cgi?f=/n/a/2005/04/10/national/w101247D57.DTL)
                            Describe Sex                                       
                            Harrassment                                        

  15         Legal          New rules drive  4/16/2005     Janice Hopkins      [[http://bmj.bmjjournals.com/cgi/content/full/330/7496/864]{.underline}](http://bmj.bmjjournals.com/cgi/content/full/330/7496/864)
                            off NIH                        Tanne               
                            researchers                                        

  16         Media          Director of NIH  8/26/2005     Ceci Connolly       [[http://www.washingtonpost.com/wp-dyn/content/article/2005/08/25/AR2005082501664.html]{.underline}](http://www.washingtonpost.com/wp-dyn/content/article/2005/08/25/AR2005082501664.html)
                            Agrees To Loosen                                   
                            Ethics Rules                                       

  17         Media          Grassley Hails   12/28/2005    Associated Press    [[http://www.honestdoctor.org/images/documents/GrassleyReinstatement.pdf]{.underline}](http://www.honestdoctor.org/images/documents/GrassleyReinstatement.pdf)
                            NIH                                                
                            Reinstatement of                                   
                            Expert                                             

  18         Media          NIH Scientist    6/14/2006     Kaisernetwork.org   [[http://www.kaisernetwork.org/daily_reports/rep_index.cfm?DR_ID=37899]{.underline}](http://www.kaisernetwork.org/daily_reports/rep_index.cfm?DR_ID=37899)
                            Violated Ethics                                    
                            Rules, Federal                                     
                            Laws,                                              
                            Congressional                                      
                            Report Finds                                       

  19         Activists      Sex                            Jyotsna Sreenivasan [[http://www.feminist.org/research/report/72_md.html]{.underline}](http://www.feminist.org/research/report/72_md.html)
                            Discrimination                                     
                            Ruins Doctor\'s                                    
                            Career                                             

  20         Whistle blower Radiation                      Jocelyn Kaiser      [[http://www.sciencemag.org/cgi/content/short/277/5334/1920a]{.underline}](http://www.sciencemag.org/cgi/content/short/277/5334/1920a)
                            Poisoning: NIH                                     
                            Case Ends With                                     
                            Mysteries                                          
                            Unsolved                                           
  ---------- -------------- ---------------- ------------- ------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

## LIST OF SELECTED RISK IQ SITES 

-   [www.riskworld.com](http://www.riskworld.com/)

-   [www.riskportal.com](http://www.riskportal.com/)

-   [www.cabinet.gov.uk](http://www.cabinet.gov.uk/)

-   www.whitehouse.gov

-   [www.bis.org](http://www.bis.org/)

-   [www.coso.org](http://www.coso.org/)

-   [www.federalreserve.gov](http://www.federalreserve.gov/)

-   [www.nist.gov](http://www.nist.gov/)

-   [www.cmu.sei.edu](http://www.cmu.sei.edu/)

-   [www.sebi.gov.in](http://www.sebi.gov.in/)

-   www.nyse.com

-   [www.euronext.com](http://www.euronext.com/)

-   [www.ogc.gov.uk](http://www.ogc.gov.uk/)

-   [www.iso.org](http://www.iso.org/)

-   [www.iodsa.co.za](http://www.iodsa.co.za/)

-   [www.tbs-sct.gc.ca](http://www.tbs-sct.gc.ca/)

-   [www.riskmanagement.com.au](http://www.riskmanagement.com.au/)

-   [www.sec.gov](http://www.sec.gov/)

-   www.bmj.bund.de

## Appendix D: Risk IQ

![](media/image36.wmf){width="4.995833333333334in"
height="3.751388888888889in"}

***Governance & Risk Oversight***

This dimension encompasses the enterprise\'s tone at the top, your
organization's risk governance structure (Oversight Body and Management
committees, charters, authorities), risk and compliance roles and
responsibilities, risk management policies including tolerance of
specific types of risk (i.e., whether or not it is willing to take
certain risks) and its appetite for those risks it is willing to take.

This dimension is the basis for all other components of enterprise risk
management and risk intelligent decision-making including your
organization's philosophy about how risk should be understood and
managed. Authority and direction is exercised by properly designated
managers over assigned resources in the accomplishment of business
objectives.

**The Oversight Body and Its Committees ("The Oversight Body")**

1.  The Oversight Body annually reviews and approves its risk management
    policy that, for example, clearly defines risk, its philosophy about
    risk, its risk tolerance, its appetite for various types of risk and
    the relationship between mission critical risks and value drivers.

2.  The Oversight Body has approved a risk management policy which
    outlines the mechanisms to delegate authorities and to elevate
    issues and conflicts.

3.  The Oversight Body reviews financial objectives of the firm
    (earnings, ROE, ROA,...) to ensure compatibility with the level of
    risk embedded in the business plan.

4.  The Oversight Body helps create an overall culture that promotes
    risk-informed decision-making at all levels of the firm.

5.  The Oversight Body monitors progress to remedy deficiencies in the
    management of mission critical risks.

6.  The Oversight Body systematically considers risk as part of its core
    decision-making processes.

7.  The Oversight Body ensures that they are updated regularly on
    mission critical risks to the enterprise and its vulnerability.

8.  Where appropriate, the Oversight Body or its Risk Management
    Committee (RMC) approves the list of traded products (futures,
    options, structured trades,...) and makes sure that the firm is
    adequately prepared to handle their risks.

9.  The Oversight Body actively supports an overall culture that
    promotes ethical and risk informed decision making at all levels of
    the enterprise.

10. The Oversight Body requires management to annually describe its
    policy and process for risk assessment and risk management for all
    risks that constitute a major financial exposure.

11. The Oversight Body has established a threshold above which all risks
    must be reported to the Oversight Body.

12. Oversight Body has received training and education on risk
    management and the responsibilities of the Oversight Body.

13. The Oversight Body has reviewed and approved the enterprise's
    standards of behavior, integrity and ethical values in a code of
    conduct.

14. The Oversight Body or the RMC monitors the action plan put in place
    to remedy deficiencies in the key risk controls and risk systems of
    the firm.

15. The Oversight Body is active in reviewing large capital commitments
    and investments.

**Responsibilities, Authorities and Accountabilities**

1.  Risk policies and procedures are well disseminated and supported by
    an effective disciplinary system.

2.  There is a comprehensive and effective Delegation of Authority
    policy that is reviewed annually by the Oversight Body.

3.  Accountability and authority for risk taking is clear defined
    throughout the organization.

4.  There is a disciplined and integrated system of risk limits for all
    businesses.

5.  Specific executives are assigned responsibility and accountability
    for the identification, assessment, prioritization and management of
    specific risks and their interactions.

**Risk Intelligence Program Strategy**

1.  A formal risk management plan has been developed -- based on
    stakeholder inputs -- to guide and communicate Risk Intelligence
    program activities.

2.  The Risk Intelligence program has defined goals and objectives,
    action plans, a budget and committed resources, in alignment with
    stated business line goals.

3.  The Risk Intelligence budget is commensurate with the risk profile
    of the enterprise.

**Roles and Responsibilities**

1.  A program management office (PMO) has been established to set the
    direction of primary Risk Intelligence activities and to promote a
    coordinated and leveraged approach to Risk Intelligence across the
    organization.

2.  Existing risk and compliance functions have harmonized, synchronized
    and rationalized their risk management activities e.g., there is a
    common language of risk and risk assessment.

3.  Senior Management is effectively engaged in the Risk Intelligence
    process, serving as a Steering Committee or other risk oversight
    council.

4.  All business units are effectively aligned with the Risk
    Intelligence program as providers of inputs (e.g. risk data),
    participants in the risk assessment and management process, and as
    users of Risk Intelligence outputs (to improve risk informed
    strategic plans, capital allocation, operations, etc.).

5.  Responsibilities are clearly defined for developing and sustaining
    risk management capabilities across the enterprise.

6.  Corporate risk management responsibility rests with an influential
    high-level officer (e.g. a Chief Risk Officer or Director of ERM)
    who reports directly to the CEO and/or the Oversight Body.

***Risk Identification***

Risk is the potential for loss of sub-optimization of gain. Management
should identify potential internal and external risks that are relevant
to the business and could significantly, adversely affect the entity and
its key objectives, projects, processes, functions and/or systems. Risks
are considered as scenarios and chains of events rather than as isolated
incidents. This includes risks to future growth objectives ("value
creation") as well as risks to existing assets ("value preservation").

1.  Strategic, operational, financial reporting, and compliance
    objectives have been defined by the organization and communicated
    appropriately.

2.  Executive management has identified the key value drivers of the
    business and the associated risks and scenarios.

3.  A taxonomy of risks relevant to the industry and the specific
    business is updated regularly based on the volatility of the
    business and its operating environment.

4.  Standardized definitions of such risks have been developed to
    promote consistency of usage and interpretation.

5.  Stress tests and sensitivity analyses are conducted to identify
    extreme cases (i.e., upside and downside).

6.  Tools and techniques (such as Failure Mode and Effects Analysis) are
    used to identify how the enterprise might fail to achieve its
    objectives in terms of both future growth as well as protecting its
    existing assets.

7.  There is a systematic and timely evaluation of both the external and
    internal environment to identify opportunities, threats strengths
    and weaknesses (SWOT).

8.  Risk interdependencies are identified to better understand the
    cumulative effect of interrelated risk exposures.

***Risk Assessment & Measurement***

Risk assessment enables the business to consider (1) the extent to which
potential events may have an impact on achievement of its objectives and
(2) the net exposure of the business after taking into account current
risk mitigation and controls. Mitigated values are determined and
reported to the Oversight Body.

1.  Risks are assessed on a consistent, enterprise-wide basis on gross
    impact (before mitigation and controls) and net impact (or
    vulnerability- after mitigation and controls).

2.  Assessments are both qualitative and quantitative and utilize
    appropriate tools, measures and techniques consistent with the type
    and complexity of risk.

3.  Mitigated value is calculated, i.e., the difference between gross
    and net risk.

4.  Internal and external subject matter specialists are involved as
    appropriate in the assessment of risk.

5.  The organization's exposure to extreme upsides and downsides is
    assessed.

6.  Probabilistic analyses are only used when appropriate, e.g., when
    there is a law of large numbers, and cause and effect relationships
    are known.

7.  Statistical risk measures are supplemented with stress-testing and
    scenario analysis.

8.  Risk measures are adapted to the types of risks taken.

9.  Root Cause Analysis is performed when significant risks do occur.

10. The risk taxonomy is revised based on emerging and experienced risks

11. The risk assessment process and criteria are harmonized across all
    enterprise functions performing risk assessments. This includes for
    example, internal audit, compliance and SOX.

12. A systematic independent verification of the risk assessment process
    and results is used to validate its relevance and efficiency.

***Response Alternatives & Response Plans***

Risk response is management\'s determination on how best to respond to a
specific risk or set of risks. This includes whether to avoid a risk,
accept it (and mitigate where possible) and/or transfer it, prioritize
the allocation of resources and then execute the plan.

1.  Alternative risk responses are formulated and evaluated for
    alignment with the entity\'s risk tolerance and risk appetite.

2.  Risk responses are organized in a hierarchy of response, i.e, from
    lowest to highest strength of response and cost.

3.  Management determines priorities taking into account such factors as
    speed of risk onset, likelihood of occurrence, urgency, cost of
    mitigation compared to expected benefit, degree of difficulty and
    time required to implement.

4.  Responses are integrated and communicated to provide effective and
    timely enterprise-wide preparation, response and recovery.

5.  Risk responses meet compliance requirements.

6.  Risk responses are aligned with corporate and financial objectives
    through the budgeting and planning process.

7.  Response plans include specific tasks, resources required,
    responsibilities and time frames.

8.  Management supports risk owners/managers with tools, experienced
    staff, venues for discussion, knowledge-sharing, and advisory
    services.

***Control Activities, Testing and Assurance***

Control activities are the policies, procedures and systems that help
ensure that management's risk responses are carried out. Control
activities occur throughout the organization, at all levels and in all
functions. They include a range of activities − as diverse as approvals,
authorizations, verifications, reconciliations, reviews of operating
performance, security of assets, and segregation of duties. Controls
must be periodically tested by independent functions to ensure they are
designed appropriately and operating as intended so that Management has
assurance to support their assertions about the control environment and
the effectiveness and efficiency of controls and control organizations.

1.  Management has implemented appropriate control activities to ensure
    that its risk responses are carried out properly and in a timely
    manner for all mission critical risks.

2.  Mandatory disclosure requirements are met.

3.  General and application controls, including preventive, detective,
    manual, computer and management controls, are clearly defined in
    policies and procedures for each mission critical risk.

4.  Control activities are matched to the type of risk and the speed of
    risk onset.

5.  Controls are appropriately tested across key processes, systems and
    functions throughout the enterprise.

6.  Deficient conditions identified as a result of assurance activities
    are promptly investigated and appropriate corrective actions are
    taken.

7.  A systematic independent verification of risk analytics is used to
    validate the relevance and efficiency of measures.

8.  Processes are in place to provide management assurance that its
    confidence in the effectiveness and efficiency of key controls is
    justified.

9.  The organization has established an Internal Audit function to
    provide monitoring assistance and assurance.

***Risk Intelligence, Communication and Training***

Risk intelligence is the product resulting from gathering and analyzing
available information concerning risks to the enterprise. Pertinent risk
intelligence is identified, captured, and communicated on a timely basis
that enables trained people to carry out their responsibilities.
Personnel are trained to make rapid and appropriate decisions using risk
intelligence provided to them.

1.  Risk Intelligence is integrated with core decision-making processes
    and risk and compliance functions.

2.  Risk-related performance metrics, dashboards, and scorecards have
    been established for all mission critical risks.

3.  Risk information systems enable managers to access and aggregate
    risk data quickly and from all mission critical locations.

4.  Effective communication occurs, flowing down, across and up the
    organization to the Oversight Body, as well as with external
    parties, such as customers, suppliers, regulators, shareholders, and
    stakeholders.

5.  The organization takes steps to communicate effectively its risk
    management standards and procedures to all employees and other
    agents, e.g., by requiring participation in training programs or by
    disseminating publications that explain in a practical manner what
    is required.

6.  The organization makes timely and pertinent disclosures to internal
    and external parties, including key stakeholders (shareholders,
    regulators, etc) and tracks the timeliness of its response to
    requests for information.

7.  Training related to Risk Intelligence and its associated processes,
    activities and expectations are carried out at all levels of the
    organization.

8.  There is a commitment to competence to ensure that all individuals
    have the necessary knowledge and skills to perform their duties.

***Monitoring and Escalation***

Monitoring is the periodic or continuous observation of the enterprise's
portfolio of risk exposures in order to detect and give timely warning
of change. Monitoring includes supervision, observation and reporting to
responsible individuals. Monitoring is an ongoing activity embedded into
the entity\'s operations. Escalation is a procedure by which risks that
exceed or are about to exceed specified thresholds or triggers are
elevated to the appropriate level of authority for resolution on a
timely basis.

1.  There is an early warning system based on established thresholds.

2.  The frequency of monitoring and reporting is sufficient to detect
    significant variations in exposure on a timely basis.

3.  Thresholds and triggers for escalation to management and the
    Oversight Body have been established for all mission critical risks.

4.  There is an escalation mechanism for risks that exceed specific
    thresholds, with associated timeframes for escalation.

5.  There is an escalation mechanism for breaches of decision
    authorities.

***Sustainability and Continuous Improvement***

Risk intelligence should be sustainable and depends on the capability of
people, processes and systems to act in an integrated, coordinated and
timely manner. Improvement of effectiveness and efficiency is always
possible, so risk management processes should be continuously
reevaluated and improvements implemented.

1.  The organization has a process to monitor its Risk Intelligence
    program to review and evaluate risk management activities compared
    to its objectives (e.g. value creation/preservation; adherence to
    applicable laws, regulations; harmonizing all key risk management
    activities).

2.  Management monitors its Risk Intelligence program effectiveness to
    review and evaluate whether or not substantial progress is being
    made in managing risk exposures and takes corrective action to
    ensure that exposures are within the enterprise's appetite for such
    exposures.

3.  There is a detailed program of improvement resulting from activities
    that may include for example: internal reviews, internal or external
    assessments, user feedback, complaints and other issues.

4.  Lessons learned are identified and communicated to appropriate
    personnel on a timely basis.

5.  Failures to correctly identify and assess risks are investigated and
    remediation is implemented.

6.  Risk related policies and procedures are reviewed and updated on a
    timely basis.

7.  Management maintains a risk intelligence knowledge base embedded in
    core enterprise management processes.

8.  Management has instituted a system of change management to promote
    timely and effective response to risk exposures.

9.  The organization has a successful track record of managing people,
    process and technology transformation.

As of July 11, 2006

## Frequently Used Terms in this Study

The Study may refer to several terms with summary meanings described
here.

**Event** -- An incident or occurrence, from sources internal or
external to an entity that affects achievement of objectives

**Failure Modes and Effects Analysis --** a structured method to
describe the way processes and systems might fail to meet performance
objectives, the consequences of the failure, current plans to mitigate
the failure, and assurance that mitigation exists, is sufficient,
effective, and efficient.

**Gross Risk** -- The risk in an organization's business or mission
prior to accounting for mitigation activities, such as risk and quality
management, insurance, controls, safety measures, etc.

**Impact** -- The risk to an entity in the absence of any actions
management might take to alter the potential consequences or impact by
addressing factors that contribute to vulnerability (i.e., controls,
insurance, and mitigation strategies).

**Inherent Risk** -- This is *Gross Risk*.

**Intolerable Risk** -- This is an organization's identification of a
risk that it does not have the authority to accept.

**Likert Scale --** (pronounced lick-ert) A type of categorical measure
using standardized response categories in survey questionnaires.
Typically a range of questions using response categories such as
strongly agree, agree, disagree, and strongly disagree, or ranging from
very low to very high, are utilized to construct a composite measure.
The scale was introduced by Rensis Likert in 1932.

**MARCI Chart** -- An example of a risk map that has two reporting
formats. In one format, risks are displayed in a scatter plot, where
gross risk is measured along the vertical axis (y-axis) and net risk is
measured along the horizontal axis (x-axis). In the figure below, risks
that are high impact on the value where vulnerability is also high
should be mitigated immediately (M) and remediation actions tracked at
the executive and program level. For risks that are deemed high impact,
but where vulnerability is considered low because of controls or
mitigation, the organization should assure (A) that the measurement of
vulnerability is realistic and the confidence in preparedness is
justified. Risks that are low impact and low vulnerability should be
tested and monitored under its normal controls environment and look for
efficiencies so that resources dedicated to low impact, low
vulnerability risk can be redeployed (R). For risks that have low
impact, but where vulnerability is high, these risks should be examined
independently, but also measured for their frequency (death by a
thousand cuts) and cumulative impact (CI) on the organization's
objectives, should multiple risks occur simultaneously.

![](media/image37.wmf){width="4.375in" height="3.6590277777777778in"}

In a second format, the MARCI chart is arranged as a bar chart. This
allows for the depiction of Mitigated Value in a sorted bar graph with a
table to rank and recommend action.

![](media/image38.wmf){width="5.997222222222222in"
height="2.6569444444444446in"}

![](media/image39.wmf){width="5.998611111111111in" height="1.39375in"}

**Mission Critical \[object\] -** any process, system, organization, or
operation that cannot tolerate intervention, compromise or shutdown
during the performance of its core function. Mission critical
environments usually support core objectives of the organization. These
environments also monitor, store, support and communicate data,
information and intelligence that cannot be lost or corrupted without
compromising their core function. A ***Mission Critical Risk*** is then
a potential loss due to failure in a mission critical activity.

**Mitigated Value** -- Mitigated Value equals Gross Risk minus Net Risk.
This is the value of management response to specific risks, including a
system of internal controls. Mitigated value may be positive (due to the
existence of effective and efficient controls), zRAM (due to the
non-existence of controls, or the existence of ineffective controls), or
negative (due to the inefficiency of controls).

**Net Risk** -- This is the organization\'s vulnerability to an inherent
threat, that is, with consideration of the management\'s response to the
threat. There are two considerations in the assessment of Net Risk. One
is the level of vulnerability that remains given management\'s
preparation of response and recovery from a potential threat, for
example, a business continuity plan for a data center outage affecting
all IC\'s. The other is the speed at which management can respond to the
threat, for example, management can only respond within 36 hours to a
data center outage affecting all IC\'s within 24 hours.

**Preparedness** -- The effectiveness of existing controls, processes,
procedures, systems, people, and other risk mitigation measures to
detect, prevent, and/or correct.

**Residual Risk** -- This is *Net Risk*.

**Risk** -- This is the possibility of loss, or sub-optimization of
gain, due to an event occurring that will have a negative impact on the
achievement of objectives.

**Risk Appetite** -- The degree to which an organization can accept a
level of risk as indicated in the organization's Delegation of
Authority.

**Risk Assessment** -- The process for risk identification and
prioritization of the organization's key business risks (operational,
financial, external/strategic, regulatory). The risk assessment links
risks to value by answering the following questions:

-   How can the enterprise fail to achieve its value objectives?

-   What would cause the enterprise to fail?

-   What would be the effects of the failure?

-   What is currently being done to prevent, detect, correct or escalate
    such failure?

-   What is our vulnerability to such failure?

-   What further actions are required to cost-effectively mitigate value
    at risk?

-   How do we get reasonable assurance existing mitigation is reliable
    and effective?

**Risk Intelligence** -- The process and the capability of gathering,
understanding, monitor, report, and respond to an organization's risks
to value. Risk intelligence capabilities are measured along dimensions
of governance and risk oversight, risk identification, risk assessment,
risk response, monitoring and escalation, control assurance and testing,
risk intelligence performance and training, and sustainable and
continuous process improvement.

**Risk IQ --** by analogy with an intelligence quotient, the Risk IQ
measures an organization's risk intelligence capability and maturity.
The Risk IQ is a five point Likert scale with 1 the lowest Risk IQ or
level of intelligence capability maturity, and 5 the highest level. Risk
intelligence capability is measured against eight dimensions of risk
intelligence: governance, risk identification, risk assessment and
measurement, risk response alternatives and plans, monitoring and
escalation, control activities, assurance and testing, risk
intelligence, communications and training, and sustainability and
continuous improvement.

**Risk Map** -- Visual representation of the organization's risk profile
in the form of a graph or table of the impact, vulnerability, and
mitigated value rankings of the risks listed in the Risk Directory, and
ranked from very high to very low. The MARCI chart and accompanying
tables is an example of a Risk Map.

**Risk Model** -- The methodology for assessing, responding to,
monitoring and escalating risks. This methodology contains
organizational, process, people, systems details; risk ownership,
subject matter specialists; risk definitions and taxonomies, impact and
vulnerability, mitigated value, and risk management capability criteria;
mitigation, assurance, redeployment, cumulative impact actions; action
plans, status and priorities; reporting requirements; by inclusion the
Risk Model contains the Process, Organizational, and Risk Segmentations.

**Risk Segmentation** -- this is the partitioning of risk into component
process and systems. Each risk has a contribution from at least one
process or system. Processes and systems are chosen since they
potentially cut across multiple organizations and objectives.

**Tolerable Risk** -- An organization's identification of a risk that it
has the authority to accept.

**Vulnerability** -- This is *Net Risk*.

[^1]: For the purposes of this study, an operational definition of
    "sound scientific design" is a set of risk assessment rules,
    definitions, process, procedures and controls that is grounded in
    scientific knowledge, both reason and experience. "Sound scientific
    design" requires the use of a logical research process, the hallmark
    of scientific inquiry that embeds criteria of logical validity and
    empirical verifiability to acquire and produce knowledge.
    (Frankfort-Nachmias and Nachmias (2000)

[^2]: By "methodology" is meant a set of rules, definitions, process,
    procedures and controls. A definition consistent with this Study is
    "a documented approach for performing activities in a coherent,
    consistent, accountable, and repeatable manner" (Treasury Enterprise
    Architecture in Interoperability ClearingHouse at
    http://www.ichnet.org/glossary.htm)

[^3]: GAO-05-207 High Risk Update, January 2005. GAO also noted for HHS:
    Risk management is a continuous process to identify, monitor, and
    mitigate risks to ensure that the risks are being properly
    controlled and that new risks are identified and resolved as early
    as possible. An effective risk management process is designed to
    mitigate the effects of undesirable events at the earliest possible
    stage to avoid costly consequences." (GAO-04-1089T)

[^4]: This may be another example of the Hawthorne Effect: the
    experimenter may observe an expected effect, but not for a reason
    hypothesized in the experiment, possibly due to the participant's
    knowledge that the experiment is being conducted. The effect is
    named after a series of experiments performed at the Hawthorne works
    of Western Electric, Chicago, Illinois during the 1930's. See Mayo
    (1933).

[^5]: [Risk intelligence]{.underline} is defined as the process and the
    capability of gathering, understanding, monitoring, reporting, and
    responding to risks to performance of NIH mission.

[^6]: These dimensions are summarized with key consideration questions
    and frame the observations and recommendations below.

[^7]: One operational definition of "sound scientific design" is a set
    of risk assessment rules, definitions, process, procedures and
    controls that is grounded in scientific knowledge. This knowledge
    includes both reason and experience. "Sound scientific design"
    requires the use of a logical research process, the hallmark of
    scientific inquiry that embeds criteria of logical validity and
    empirical verifiability to acquire and produce knowledge.
    (Frankfort-Nachmias and Nachmias (2000))

[^8]: "Methodology" includes a set of rules, definitions, process, and
    procedures. A definition consistent with this Study is "a documented
    approach for performing activities in a coherent, consistent,
    accountable, and repeatable manner" (Treasury Enterprise
    Architecture in Interoperability ClearingHouse at
    http://www.ichnet.org/glossary.htm).

[^9]:

[^10]: GAO-05-207 High Risk Update, January 2005. GAO also noted for
    HHS: Risk management is a continuous process to identify, monitor,
    and mitigate risks to ensure that the risks are being properly
    controlled and that new risks are identified and resolved as early
    as possible. An effective risk management process is designed to
    mitigate the effects of undesirable events at the earliest possible
    stage to avoid costly consequences." (GAO-04-1089T)

[^11]: This list of constraints is developed from Kontio et al. (1996).

[^12]: These constraints were based on the empirical work of Kontio et
    al. (1996) in developing software engineering project risk
    management methodology.

[^13]: See Miles and Huberman (1994) and Trochim (2001) for a comparison
    of various qualitative data analytical techniques.

[^14]: [Risk intelligence]{.underline} is defined as the process and the
    capability of gathering, understanding, monitoring, reporting, and
    responding to risks to performance of NIH mission.

[^15]: For example, see Aamodt and Plaza (1994): " Instead of relying
    solely on general knowledge of a problem domain, or making
    associations along generalized relationships between problem
    descriptors and conclusions, \[Case Based Reasoning, CBR\] is able
    to utilize the *specific* knowledge of previously experienced,
    concrete problem situations (cases). A new problem is solved by
    finding a similar past case, and reusing it in the new problem
    situation. A second important difference is that CBR also is an
    approach to incremental, sustained learning, since a new experience
    is retained each time a problem has been solved, making it
    immediately available for future problems. The CBR field has grown
    rapidly over the last few years, as seen by its increased share of
    papers at major conferences, available commercial tools, and
    successful applications in daily use."

[^16]: "The risk exposure is different depending on the company and
    particular facts and circumstances surrounding that entity. It seems
    reasonable that the risk factors \[you\] disclose in your financial
    statements should be included in the risk analysis. Whether the
    analysis is sufficient if limited to those items is something the
    audit committee would have to determine." Janice O\'Neill, Senior
    Vice President Corporate Compliance, New York Stock Exchange, Email
    communication, March 16, 2006.

[^17]: Keeney and Raiffa (1976).

[^18]: Fischhoff et al. (2000).

[^19]: Lash (2000).

[^20]: Loewenstein and Lerner (2003) and Loewenstein, Weber, Hsee and
    Welch (2001).

[^21]: See especially Tversky, A. and D. Kahneman (1974, 1981) and Russo
    and Shoemaker (1989).

[^22]: See Tversky and Kahneman (1992) , Hammond, Keeney, and Raifa,
    (2005) and Nutt (2002).

[^23]: See Wynne (1996).

[^24]: Davis and Holt (1993) and Plott (1996).

[^25]: Camerer and Weber (1992) and Camerer (1997); Keeney and Raiffa (.

[^26]: See, *The International Standards for the Professional Practice
    of Internal Auditing* (*Standards*), IIA: The chief audit executive
    should establish risk-based plans to determine the priorities of the
    internal audit activity, consistent with the organization\'s goals.
    **2010.A1** - The internal audit activity\'s plan of engagements
    should be based on a risk assessment, undertaken at least annually.
    The input of senior management and the board should be considered in
    this process **2010.C1** - The chief audit executive should consider
    accepting proposed consulting engagements based on the engagement\'s
    potential to improve management of risks, add value, and improve the
    organization's operations.  Those engagements that have been
    accepted should be included in the plan. **2110 -- Risk Management**
    The internal audit activity should assist the organization by
    identifying and evaluating significant exposures to risk and
    contributing to the improvement of risk management and control
    systems. **2110.A1** - The internal audit activity should monitor
    and evaluate the effectiveness of the organization\'s risk
    management system. **2110.A2** - The internal audit activity should
    evaluate risk exposures relating to the organization\'s governance,
    operations, and information systems regarding the

    Reliability and integrity of financial and operational information.

    Effectiveness and efficiency of operations.

    Safeguarding of assets.

    Compliance with laws, regulations, and contracts.

    **2110.C1 -** During consulting engagements, internal auditors
    should address risk consistent with the engagement's objectives and
      be alert to the existence of other significant risks**. 2110.C2**
    -- Internal auditors should incorporate knowledge of risks gained
    from consulting engagements into the process of identifying and
    evaluating significant risk exposures of the organization.

[^27]: **[Risk Intelligence]{.underline}** is the process and the
    capability of gathering, understanding, monitor, report, and respond
    to a organization's risks to value. Risk intelligence capabilities
    are measured along dimensions of governance and risk oversight, risk
    identification, risk assessment, risk response, monitoring and
    escalation, control assurance and testing, risk intelligence
    performance and training, and sustainable and continuous process
    improvement.

[^28]: These categories are based on Deloitte & Touche LLP proprietary
    evergreen survey and analysis of disclosed risks housed in its
    Publicly Reported Risk Repository (PRRR). As of June 30, 2006, the
    publicly available risks of over 200 organizations are contained in
    the PRRR. Sources considered include SEC 10K and 20F filings, and
    publicly reported risks, losses, incidents, and other open source
    information about risk events and their impact on organizations.

[^29]: See Kaplan and Norton, Strategy Maps.

[^30]: See Philip Niven, Balanced Scorecard for Public Sector
    Organizations,

[^31]: By mission critical is meant any activity that is essential to
    the strategy, operations, infrastructure, or stakeholders relations
    of an organization.

[^32]: NIST SP 800-30 provides a referential link between managing IT
    asset risk and managing the risk of an agency overall: "An effective
    risk management process is an important component of a successful IT
    security program. The principal goal of an organization's risk
    management process should be to protect the *organization and its
    ability to perform their mission*, not just its IT assets.
    Therefore, the risk management process should not be treated
    primarily as a technical function carried out by the IT experts who
    operate and manage the IT system, but as an essential management
    function of the organization." (p. 1)

[^33]: The Likert scale is the primary assessment measuring tool. It
    usually has an odd number of categories against which data can be
    anchored. More detail in provided in the sections on definition and
    procedures below

[^34]: By NBS is meant the NIH Business System along with any legacy
    system improvements, and other best in class system implementations
    to automate various processes across the NIH.