Coast Guard: Security Risk Model Meets DHS Criteria, but More Training Could Enhance Its Use for Managing Programs and Operations
Released: 2011-12-19T00:00:00

Letter

Background

Risk Management

In recent years, we, Congress, the 9/11 Commission, and others have recommended that federal agencies with homeland security responsibilities utilize a risk management approach to help ensure that finite resources are dedicated to assets or activities considered to have the highest security priority. The purpose of risk management is not to eliminate all risks, as that is an impossible task. Rather, given limited resources, risk management is a structured means of making informed trade-offs and choices about how to use available resources effectively and monitoring the effect of those choices. Thus, risk management is a continuous process that includes the assessment of threats, vulnerabilities, and consequences to determine what actions should be taken to reduce or eliminate one or more of these elements of risk.

To provide guidance to agency decision makers, we developed a risk management framework, which is intended to be a starting point for applying risk-informed principles. Our risk management framework entails a continuous process of managing risk through a series of actions, including setting strategic goals and objectives, assessing risk, evaluating alternatives, selecting initiatives to undertake, and implementing and monitoring those initiatives. Additional information on risk management, including our risk management framework, can be found in appendix I.

DHS is required by statute to utilize risk management principles with respect to various DHS functions. With regard to the Coast Guard, federal statutes call for the Coast Guard to use risk management in specific aspects of its homeland security efforts. The Maritime Transportation Security Act of 2002 (MTSA), for example, calls for the Coast Guard and other port security stakeholders, through implementing regulations, to carry out certain risk-based tasks, including assessing risks and developing security plans for ports, facilities, and vessels. In addition, the Coast Guard Authorization Act of 2010 requires, for example, the Coast Guard to (1) develop and utilize a national standard and formula for prioritizing and addressing assessed security risks at U.S. ports and facilities, such as MSRAM; (2) require Area Maritime Security Committees to use this standard to regularly evaluate each port’s assessed risk and prioritize how to mitigate the most significant risks; and (3) make MSRAM available, in an unclassified version, on a limited basis to regulated vessels and facilities to conduct risk assessments of their own facilities and vessels.

Coast Guard Security Risk Models

From 2001 to 2006, the Coast Guard assessed maritime security risk using the Port Security Risk Assessment Tool (PSRAT), which was quickly developed and fielded after the terrorist attacks of September 11, 2001. PSRAT served as a rudimentary risk calculator that ranked maritime critical infrastructure and key resources (MCIKR) with respect to the consequences of a terrorist attack and evaluated vessels and facilities that posed a high risk of a transportation security incident. While PSRAT provided a relative risk of targets within a port region, it could not compare and prioritize relative risks of various infrastructures across ports, among other limitations.

Recognizing the shortcomings of PSRAT that had been identified by the Coast Guard and us, in 2005 the Coast Guard developed and implemented MSRAM to provide a more robust and defensible terrorism risk analysis process. MSRAM is a risk-based decision support tool designed to help the Coast Guard assess and manage maritime security risks throughout the Coast Guard’s area of responsibility. Coast Guard units throughout the country use this tool to assess security risks to over 28,000 key maritime infrastructure assets—also known as targets—such as chemical facilities, passenger terminals, and bridges, as well as vessels such as cruise ships, ferries, and vessels carrying hazardous cargoes, among other things. Unlike PSRAT, MSRAM is designed to capture the security risks facing different types of targets, allowing comparison between different targets and geographic areas at the local, regional, and national levels. MSRAM’s risk assessment methodology assesses the risk of a terrorist attack based on different scenarios; that is, it combines potential targets with different attack modes for each target/attack mode combination (see table 1). MSRAM automatically determines which attack modes are required to be assessed for each target type, though local MSRAM analysts have the ability to evaluate additional optional attack modes against any target. For each target/attack mode combination, MSRAM can provide different risk results, such as the inherent risk of a target and the amount of risk mitigated by Coast Guard security efforts.

MSRAM calculates risk using the following risk equation: Risk = Threat x Vulnerability x Consequence. Numerical values representing Coast Guard’s assessment of threat (or relative likelihood of attack), vulnerability should an attack occur, and consequences of a successful attack are combined to yield a risk score for each maritime target. The model calculates risk using threat judgments provided by the Coast Guard Intelligence Coordination Center (ICC), and vulnerability and consequence judgments provided by MSRAM users at the sector level— typically Coast Guard port security specialists—which are reviewed at the district, area, and headquarters levels. The risk equation variables are as follows: Threat represents the relative likelihood of an attempted attack on a target. The ICC provides threat probabilities to MSRAM, based upon judgments regarding specific intent, capability, and geographic preference of terrorist organizations to deliver an attack on a specific type of maritime target class—for example, a boat bomb attack on a ferry terminal. To make these judgments, ICC officials use intelligence reports generated throughout the broader intelligence community to make qualitative determinations about certain terrorist organizations and the threat they pose to the maritime domain. At the sector level, Coast Guard MSRAM users do not input threat probabilities and are required to use the threat probabilities provided by the ICC. This approach is intended to ensure that threat information is consistently applied across ports.

Vulnerability represents the probability of a successful attack given an attempt. MSRAM users at the sector level assess the vulnerability of targets within their respective areas of responsibility. Table 2 shows the factors included in the MSRAM vulnerability assessment.

Consequence represents the projected overall impact of a successful attack on a given target or asset. Similar to vulnerability assessments, MSRAM users at the sector level assess the consequences of a successful attack on targets within their respective area of responsibility. Table 3 shows the factors included in the MSRAM consequence assessment.

In addition to the consequence factors listed in table 3, sector MSRAM users also assess the response capabilities of the Coast Guard, port stakeholders, and other governmental agencies and their ability to mitigate death/injury, primary economic, and environmental consequences of a successful attack. Because there is a broad array of target types operating in the maritime domain that can result in different types of impacts if successfully attacked, MSRAM uses an approach for drawing equivalencies between the different types of impacts. This approach was based on establishing a common unit of measure, called a consequence point. One consequence point represents $1 million of equivalent loss to the American public.

To support MSRAM development and risk analysis at the headquarters level, the Coast Guard has provided MSRAM-dedicated staff and resources. According to the Coast Guard, resources for MSRAM or port security risk analysis are not from a specific budget line item. From fiscal year 2006 to fiscal year 2011, the Coast Guard reported assigning from two to five staff (full-time equivalents) and from $0.6 million to $1.0 million annually to support MSRAM at headquarters. There are no MSRAM- dedicated staff at the area, district, and sector levels; rather, MSRAM assessment and analysis is generally conducted by port security specialists, who have other responsibilities. The port security specialist typically has responsibility for numerous activities, including the Port Security Grant Program, Area Maritime Security Committees, and Area Maritime Security Training Exercise Program, among others.

DHS Risk Management Criteria

The NIPP is DHS’s primary guidance document for conducting risk assessments and includes core criteria that identify the characteristics and information needed to produce quality risk assessment results. The NIPP’s basic analytical principles state that risk assessments should be complete, reproducible, documented, and defensible, as defined in table 4.

MSRAM Risk Assessments Generally Align with DHS Criteria, but Challenges Remain

MSRAM generally aligns with DHS’s criteria for a complete and reproducible risk assessment, but some challenges remain, such as the limited time for Coast Guard personnel to complete assessments. MSRAM also generally aligns with the NIPP criteria for a documented and defensible risk assessment, but the Coast Guard could improve its documentation of the model’s assumptions and other sources of uncertainty, such as the subjective judgments made by Coast Guard analysts about vulnerabilities and consequences, and how these assumptions and other sources of uncertainty affect MSRAM’s results. In addition to providing decision makers with an understanding of how to interpret any uncertainty in MSRAM’s risk estimates, greater transparency and documentation could facilitate periodic peer reviews of the model—a best practice in risk management.

MSRAM Generally Aligns with NIPP Criteria for a Complete and Reproducible Risk Assessment

MSRAM generally aligns with NIPP criteria for a complete risk assessment. In accordance with NIPP criteria for a complete risk assessment, MSRAM assesses risk using three main variables— consequence, vulnerability, and threat. MSRAM’s risk assessment methodology also follows the NIPP criteria for factors that should be assessed in each of the three risk variables. Specifically, for threat, MSRAM generally follows the NIPP criteria by identifying attack methods that may be employed and by considering the adversary’s intent and capability to attack a target. MSRAM generally follows the vulnerability assessment criteria by estimating the likelihood of an adversary’s success for each attack scenario and describing the protective measures in place, and MSRAM generally follows the consequence assessment criteria by estimating economic loss in dollars, estimating fatalities, and describing psychological impacts, among other things.

MSRAM’s risk assessment methodology also generally aligns with the NIPP criteria for a reproducible risk assessment. To be reproducible, the methodology must produce comparable, repeatable results and minimize the number and impact of subjective judgments, among other things. Although Coast Guard officials acknowledge that MSRAM risk data are inherently subjective, the MSRAM model and data collection processes include features designed to produce comparable, repeatable results across sectors. For instance, the Coast Guard prepopulates threat data into MSRAM from the Coast Guard’s ICC. This allows for nationally vetted threat scores that do not rely on multiple subjective local judgments. DHS, in its 2010 Transportation Systems Sector-Specific Plan, stated that MSRAM produces comparable, repeatable results.

The Coast Guard has taken numerous actions that contribute to MSRAM being a complete and reproducible risk assessment model. To improve the quality and accuracy of MSRAM data and reduce the amount of subjectivity in the MSRAM process, the Coast Guard conducts an annual review and validation of MSRAM data produced at each sector; provides MSRAM users with tools, calculators, and benchmarks to assist in calculating consequence and vulnerability; and provides training to sectors on how to enter data into MSRAM. Specific actions are detailed below.

Annual validation and review. The Coast Guard uses a multilevel annual validation and review process, which helps to ensure that MSRAM risk data are comparable and repeatable across sectors. According to a 2010 review of MSRAM, conducting a thorough review process across sectors is especially important if the data are to be used for national-level decision making. This process includes sector, district, area, and headquarters officials and aims to normalize MSRAM data by establishing national averages of risk scores for attack modes and targets and by identifying outliers. The annual MSRAM validation and review process begins with sectors completing vulnerability and consequence assessments for targets within their areas of responsibility. Once the sector Captain of the Port validates the assessments, the risk assessment data are sent to district and area officials for review. Following these reviews, Coast Guard headquarters officials combine each sector’s data into a national classified dataset and perform a statistical analysis of the data. The statistical analysis involves calculating national averages for vulnerability, consequence, and response When determining whether a sector’s risk score capabilities risk scores.for a specific target is questionable or is an outlier, reviewers consider the results of the statistical analysis as well as supporting comments or rationale provided by sector officials.

According to the Coast Guard, for each outlier identified during the national review process, sector officials reconsider the data point in question and either change the inputs to reflect national averages or provide additional justification for why the risk score for the target in question should be outside of the national average. Headquarters officials explained that they generally accept justification for data outliers and that a goal of the review process is to spur discussions related to maritime risk rather than forcing compliance with national data averages. For example, officials from one sector told us that a small port in their sector is critical for their state’s energy imports, and accordingly, the port infrastructure is high risk on a national scale. The officials said that Coast Guard headquarters officials have questioned the relatively high risk rankings of the port’s infrastructure because they are statistical outliers, but have deferred to the expertise of the sector regarding the risk scores.

Tools and calculators. Recognizing that sector port security specialists who assess risk using MSRAM generally do not have expertise in all aspects of assessing vulnerability and consequence, the Coast Guard has regularly added new tools and calculators to MSRAM to improve the quality, accuracy, and consistency of vulnerability and consequence assessments. For example, MSRAM now includes a blast calculator that allows users to more easily determine the death and injury consequences of an explosion close to population centers. Coast Guard officials from 29 sectors (82 percent of sectors) cited a variety of challenges with assessing vulnerability and consequence values in MSRAM, but officials from 10 sectors said that it was becoming easier to do over time and officials from 14 sectors said that the tools and calculators in MSRAM have helped.

Benchmarks and recommended ranges. To limit inconsistencies caused by different judgments by individual MSRAM users and to minimize user subjectivity, the Coast Guard built into MSRAM a suggested range of scores for each risk factor—including vulnerability, consequence, and response capabilities—as well as averages, or benchmarks, of scores for each factor. The benchmarks are based on Coast Guard and expert evaluation of target classes and attack modes. The benchmarks and recommended ranges are reviewed and updated each year following the annual data revalidation cycle.

Training. The Coast Guard has also provided annual training for MSRAM users, including beginning, intermediate, and advanced courses intended to standardize the data entry process across the Coast Guard. Officials from 34 sectors (97 percent) reported finding the training moderately to very useful in terms of enhancing their ability to assess, understand, and communicate the risks facing their sectors. In 2011, Coast Guard headquarters also started providing live web-based training sessions on various MSRAM issues, such as resolving national review comments, to help sector staff gain familiarity with MSRAM’s features on an as-needed basis. In addition to MSRAM training provided by headquarters, one Coast Guard district official we spoke with had developed and provided localized training to the sector-level port security specialists on assessing the vulnerability of chemical facilities. The district official told us that Coast Guard headquarters was interested in this local model for delivering training and was planning to pilot a similar training program in a different district.

MSRAM generally aligns with DHS’s criteria for a complete and reproducible risk assessment, but challenges remain with the MSRAM methodology and risk assessment process. The Coast Guard has acknowledged these challenges and limitations and has actions underway to address them and make MSRAM more complete and reproducible. Coast Guard officials noted that some of these challenges are not unique to MSRAM and are faced by others in the homeland security risk assessment community. Specific challenges are detailed below.

Data subjectivity. While the Coast Guard has taken actions to minimize the subjectivity of MSRAM data, officials acknowledged that assessing threat, vulnerability, and consequence is inherently subjective. To assess threat, the Coast Guard’s ICC quantifies judgments related to the intent and capability of terrorist organizations to attack domestic maritime infrastructure. However, there are limited national historic data for domestic maritime attacks and thus intelligence officials must make a number of subjective judgments and draw inferences from international maritime attacks. Further, GAO has previously reported on the inherently difficult nature of assessing the capability and intent of terrorist groups.Vulnerability and consequence assessments in MSRAM are also inherently subjective. For example, officials from 20 sectors we interviewed said that even with training, tools, and calculators, assessing consequences can be challenging and that it often involved subjectivity and uncertainty. Officials noted that assessing economic impacts—both primary and secondary—was particularly challenging because it required some level of expertise in economics—such as supply chains and industry recoverability—which port security specialists said is often beyond their skills and training. The input for secondary economic impacts can have a substantial effect on how MSRAM’s output ranks a target relative to other potential targets. Undervaluing secondary economic impacts could result in a lower relative risk ranking that underestimates the security risk to a target, or inversely, overvaluing secondary economic impacts could result in overestimating the security risk to a target. Recognizing the challenges with assessing secondary economic impacts, Coast Guard officials said they are working with the DHS Office of Risk Management and Analysis to study ways to more accurately assess secondary economic impacts. Additionally, during the course of our review the Coast Guard implemented a tool called IMPLAN that has the potential to inform judgments of secondary economic impacts by showing what the impact could be for different terrorist scenarios.

Limited time to complete assessments. Officials from 19 sectors (54 percent) told us that the lack of time to complete their annually required vulnerability and consequence assessments is a key challenge and many expressed that they believed their sector’s data suffered in quality as a result. Each year, sectors are required to update and validate their risk assessments for targets in their areas of responsibility, which can involve site visits to port facilities and discussions with facility security officers to obtain information on vulnerability and consequences. Officials from a Gulf Coast sector noted that obtaining this information from facilities can be challenging because of the number of facilities in the sector and the time involved in meeting with each facility. Officials from an inland river sector also noted that gathering data from certain facilities—such as information on a chemical plant’s security enhancements or the expected loss of life from a terrorist attack—is challenging because facilities may not want to share proprietary information that could be damaging in the hands of a competitor. As a result, it often takes additional visits, phone calls, e-mails, and time to obtain this information. Officials from a northeastern sector said that having the people and time to update MSRAM data is their key challenge and completing the update is a heavy lift because the update is required at the same time as several other requirements, such as reviewing investment justifications for the Port Security Grant Program. Coast Guard sector officials and one district official we spoke with reported raising concerns to headquarters about the time it takes to complete MSRAM assessments. Headquarters staff also said they were looking into additional ways to make the assessment process easier for sectors, such as providing job aids and examining the possibility of completing the data update at different times in the year.

Limitations in modeling methodology—adaptive terrorist behavior. There are inherent limitations in the overall methodology the Coast Guard uses to model risk. For instance, MSRAM threat information does not account for adaptive terrorist behavior, which is defined by the National Research Council as an adversary adapting to the perceived defenses around targets and redirecting attacks to achieve its goals. Accounting for adaptive terrorist behavior could be modeled by making threat a function of vulnerability and consequence rather than the MSRAM formula which treats threat, vulnerability, and consequence as independent variables.a critique of MSRAM raised by terrorism risk assessment experts. For example, officials from the DHS Office of Risk Management and Analysis have stressed the need to account for adaptive terrorist behavior in risk models. In addition, DHS’s 2011 Risk Management Fundamentals guidance states that analysts should be careful when calculating risk by multiplying threats, vulnerabilities, and consequences (as MSRAM does), Not accounting for adaptive terrorist behavior is especially for terrorism, because of interdependencies between the three variables. Coast Guard officials agreed with the importance of accounting for adaptive terrorist behavior and with the risks of treating threat, vulnerability, and consequence as independent variables. The officials explained that although they did not design MSRAM to account for adaptive terrorist behavior, they are working to develop the Dynamic Risk Management Model, which will potentially address this issue.

For more information on network effects, see Gerald G. Brown, W. Matthew Carlyle, Javier Salmerón, and Kevin Wood, Operations Research Department, Naval Postgraduate School, Analyzing the Vulnerability of Critical Infrastructure to Attack and Planning Defenses (Monterey, Calif.: 2005). initiatives to identify and document networked systems of targets that if successfully attacked would have large ripple effects throughout the port or local economy. Coast Guard officials agreed that assessing network effects is a challenge and they are examining ways to meet this challenge. However, the Coast Guard’s work in this area is still in its infancy and there is uncertainty regarding the way in which the agency will move forward in measuring network effects.

Additional Documentation Could Align MSRAM with NIPP Criteria for Documented and Defensible Risk Assessments

MSRAM is generally documented and defensible, but the Coast Guard could improve its documentation of the model’s assumptions and other sources of uncertainty, such as subjective judgments made by Coast Guard analysts about threats, vulnerabilities, and consequences, and how these assumptions and other sources of uncertainty affect MSRAM’s results. The NIPP states that for a risk assessment methodology to be documented, any assumptions and subjective judgments need to be transparent to the individuals who are expected to use the results. For a risk assessment methodology to be defensible, uncertainty associated with consequence estimates and the level of confidence in the vulnerability and threat estimates should also be communicated to users of the results.

There are multiple assumptions and other sources of uncertainty in MSRAM. For example, assumptions used in MSRAM include the particular dollar value for a statistical life or the assumed dollar amount of environmental damage resulting from oil or hazardous material spilled as the result of a terrorist attack. MSRAM also relies on multiple subjective judgments made by Coast Guard analysts, which mean a range of possible values for risk calculated from the model. For example, to assess risk in MSRAM, Coast Guard analysts make judgments regarding such factors as the likelihood of success in interdicting an attack and the number of casualties expected to result from an attack. These subjective judgments are sources of uncertainty with implications that, according to the NIPP and risk management best practices, should be documented and communicated to decision makers.

MSRAM’s primary sources of documentation provide information on how data are used to generate a risk estimate and information on some assumptions, and the Coast Guard has made efforts to document and reduce the number of assumptions made by the field-level user in order to increase the consistency of MSRAM’s data. For example, the MSRAM training and software manual states that MSRAM users are expected to specify the assumptions they make in evaluating various attack modes and provides assumptions for users to consider when scoring attack scenarios, such as specifying the type and amount of biological agent used in a biological attack scenario and assuming that attackers are armed and suicidal in a boat bomb attack scenario.

While these documentation efforts are positive steps to reduce MSRAM data subjectivity and increase data consistency, we found that the Coast Guard has not documented all the sources of uncertainty associated with threat, vulnerability, and consequence assessments and what implications this uncertainty has for interpreting the results, such as an identification of the highest-risk targets in a port. As a result, decision makers do not know how robust the risk rankings of targets are and the degree to which a list of high-risk targets could change given the uncertainty in the risk model’s inputs and parameters. Moreover, overlapping ranges of possible risk values caused by uncertainty could have implications for strategic decisions or resource allocation, such as allocating grant funding or targeting patrols. Overlapping ranges of risk values due to uncertainty also underscores the importance of professional judgment in decision making because risk models do not produce precise outcomes that should be followed without a degree of judgment and expertise.

According to the NIPP, the best way to communicate uncertainty will depend on the factors that make the outcome uncertain, as well as the amount and type of information that is available. The NIPP states that in any given terrorist attack scenario there is often a range of outcomes that could occur, such as a range of dollar amounts for environmental damage or a range of values for a statistical life. For some incidents, the range of outcomes is small and a single estimate may provide sufficient data to inform decisions. However, if the range of outcomes is large, the scenario may require additional specificity about conditions to obtain appropriate estimates of the outcomes. Often, this means providing a range of possible outcomes rather than a single point estimate. Coast Guard officials agreed with the importance of documenting and communicating the sources and implications of uncertainty for MSRAM’s risk estimates, and noted that they planned to develop this documentation as part of an internal MSRAM verification, validation, and accreditation (VV&A) process that they expect to complete in the fall of 2011. According to the Coast Guard, accreditation is an official determination that a model or simulation is acceptable to use for a specific purpose. While this accreditation process is expected to document the scope and limitations of MSRAM’s capabilities and determine whether these capabilities are appropriate for MSRAM’s current use, the Coast Guard’s draft accreditation plan does not discuss how the Coast Guard plans to assess and document uncertainty in its model or communicate those results to decision makers.

National Research Council of the National Academies, Review of the Department of Homeland Security’s Approach to Risk Analysis. addressed and should address the structure of the model, the types and certainty of the data, and how the model is intended to be used. Peer reviews can also identify areas for improvement and can facilitate sharing best practices. As we have previously reported, external peer reviews cannot ensure the success of a model, but they can increase the probability of success by improving the technical quality of projects and the credibility of the decision-making process. MSRAM has been reviewed twice—in 2010 by risk experts affiliated with the Naval Postgraduate School and, to a lesser extent, in 2009 by CREATE at the University of Southern California. The authors of the Naval Postgraduate School report stated that their review was intended to validate and verify the equations used in MSRAM, evaluate MSRAM’s quality control procedures, and review the use of MSRAM outputs to manage risk. The authors of the CREATE report stated that their review focused on suggestions for improvement rather than a comprehensive evaluation, and they suggested that the Coast Guard continue to seek feedback and reviews from the risk and decision analysis community, as well as from practitioners of other disciplines. Coast Guard officials told us that they have generally benefited from reviews of MSRAM and have worked to implement many of the resulting recommendations. Officials noted they intend to pursue external reviews of MSRAM as part of the ongoing VV&A process, but they have not identified who would be conducting the reviews, or when the reviews would occur. As the Coast Guard’s risk assessment model continues to evolve, the Coast Guard could benefit from periodic external peer review to ensure that the structure and outputs of the model are appropriate for its given uses and to identify possible areas for improvement.

Coast Guard Has Used a Risk-Informed Approach to Manage Maritime Security Risk, but Challenges Hinder Sector Efforts

MSRAM Informs Several National-Level Risk Management Efforts

MSRAM is a security risk analysis and risk management tool and the Coast Guard intends for it to be used to inform risk management decisions at all levels of command. As such, in a May 2011 guidance document, the Coast Guard set expectations for how MSRAM should be used at the national and sector levels. At the national level, the Coast Guard expects its offices to use MSRAM to support strategic plans, policy, and guidance; to integrate MSRAM into maritime security programs; and to ensure that sectors have adequate personnel ready to perform MSRAM duties, among other goals.

Operational activities include conducting boat escorts, implementing positive control measures—that is, stationing armed Coast Guard personnel in key locations aboard a vessel to ensure that the operator maintains control—and providing a security presence through various actions. (MSRO) program. By identifying the nation’s highest-risk maritime targets, MSRAM helps establish the national MCIKR list, which sectors use to complete their annually required number of MCIKR visits. According to Coast Guard officials, MSRAM has aided in reducing the MCIKR list from 740 assets to 324 assets and allowed the Coast Guard to further prioritize within that more focused list of 324, since MSRAM analysis demonstrated that a small number of assets make up the majority of the nation’s risk.

MSRAM has also been used as a tool to inform resource allocation and performance measurement, which is consistent with the Coast Guard’s goals for MSRAM. For instance, risk-informed methods and processes or models, such as MSRAM, are used in the Coast Guard’s annual Standard Operational Planning Process, which establishes a standardized process to apportion major assets, such as boats, aircraft, and deployable specialized forces. Coast Guard officials said that MSRAM data supports the PWCS mission in this process by demonstrating how risk is distributed geographically. In addition, Coast Guard used MSRAM to support a funding request for boats, personnel, and associated support costs to assist with Coast Guard efforts to reduce the risk of certain dangerous cargoes by escorting ships passing through coastal ports carrying cargoes such as liquefied natural gas. MSRAM also supports resource allocation through the Port Security Grant Program by informing the risk formula used by DHS to allocate grant funding.

MSRAM data are also used in the Coast Guard’s model for measuring its performance in the PWCS mission, which is discussed in depth later in this report.

MSRAM has also supported strategic documents and efforts throughout DHS. Specifically, the Coast Guard reported that MSRAM data are an essential building block for a number of key strategic documents, such as the National Maritime Strategic Risk Assessment, the National Maritime Terrorism Threat Assessment, and the Combating Marine Terrorism Strategic and Performance Plan, among others. In addition, the Coast Guard uses MSRAM, among other inputs, to provide DHS with maritime risk information for the Transportation Sector Security Risk Assessment tool. DHS also reported that the Coast Guard has shared MSRAM- based identification of critical assets beyond the transportation system with 13 of the 18 DHS critical infrastructure and key resource sectors.

For example, MSRAM has been used to assess the risk of some chemical facilities and power plants.

MSRAM Has Informed Some Local-Level Risk Management Efforts, but Its Use Has Been Limited by Several Factors

MSRAM has been used to inform a variety of efforts at the sector level, such as strategic planning, communication with port stakeholders, and operational and tactical decision making, but its use for operational and tactical risk management efforts has been limited by a lack of staff time, the complexity of the MSRAM tool, and competing mission demands, among other factors. The Coast Guard expects its 35 sectors, with support from its nine districts, to integrate MSRAM data into strategic, operational, and tactical plans, operations, and programs as necessary and required, among other actions.

Based on results from our interviews with officials from all 35 Coast Guard sectors, officials from 26 sectors (74 percent) reported finding MSRAM moderately to very useful for informing strategic planning, which includes developing portions of local Area Maritime Security Plans and planning security exercises. sector reported using MSRAM to find the highest-risk areas in which to conduct exercises. Further, lessons learned from the exercises are incorporated into strategic plans, which officials said leads to planning process improvements and overall better plans. However, officials from a southeastern sector pointed out that MSRAM is a snapshot view of port risk and therefore long-term strategic plans require additional information from many sources.

Area Maritime Security Plans have been established pursuant to the Maritime Transportation Security Act of 2002. Content requirements for the plans were established by 33 C.F.R. § 103.505 and expanded by the Security and Accountability For Every Port (SAFE Port) Act of 2006 to include a Salvage Response Plan. The plans are intended to sponsor and support engagement with port community stakeholders to develop, test, and when necessary, implement joint efforts for responding to and mitigating the effects of a maritime transportation security incident. percent) said that MSRAM was moderately to very useful. For instance, officials from a southeastern sector said that MSRAM is used to communicate and justify additional security procedures. Further, during annual compliance inspections, MSRAM data are discussed with facility security officers and compared to security data that the facility security officers have calculated. In addition, officials from a Gulf Coast sector reported that MSRAM provides a convenient, objective way to communicate risk to port security stakeholders, and stakeholders appreciate that risk information from MSRAM is computer driven and based on a rigorous process.

For informing sector operational and tactical decision making, such as planning MSRO activities, developing local critical infrastructure lists, and planning for special events, officials from 18 sectors (51 percent) reported that MSRAM moderately or greatly provided them with the information needed to make risk-informed decisions regarding port security. Regarding planning MSRO activities, one eastern sector reported that MSRAM was very helpful for identifying priority targets for MSRO patrols and escorts. Regarding developing local critical infrastructure lists, officials from an eastern sector said that since the sector has no assets on the national MCIKR list, they were able to use MSRAM to generate a local list to help determine patrols and other security efforts. Regarding special event planning, officials from 16 sectors (45 percent) told us they used MSRAM to determine where to allocate resources for special events, such as the Fourth of July, dignitary visits, or political conventions. For example, officials from an inland river sector said that they used MSRAM to identify possible attack scenarios and to help identify what security resources they should request to provide security for a special event. See figure 1 for photographs of various Coast Guard security-related activities that can be informed by MSRAM. In addition to using MSRAM to inform maritime security decisions, officials from almost every sector noted that they also assess and manage risk using other tools or methods, such as the High Interest Vessel matrix, outreach to port partners, working relationships with Area Maritime Security Committees, or professional judgment.

Although officials from most sectors found that MSRAM provided useful risk information for sector-level decision making, officials from 32 sectors (91 percent) reported that their overall use of MSRAM data in managing risk was hindered by a lack of staff time for data analysis, the complexity of the MSRAM tool, or competing mission demands, among other things. These challenges are discussed below.

Limited staff time for analyzing and using MSRAM. Officials from 21 sectors (60 percent) told us that limited staff time posed a challenge to incorporating MSRAM into strategic, operational, and tactical planning efforts. For example, officials from a northeastern sector said that a lack of available staff time was one of the most significant limitations to utilizing MSRAM. These officials stated that they would like to have dedicated MSRAM personnel to develop the tool and make it useful on a daily basis. They added that even though MSRAM had many capabilities, they were unable to use it to its full capability because their port security specialist—the primary user of MSRAM—was busy with other programs, such as the Port Security Grant Program. Each of the port security specialists from the three districts we interviewed—which encompass 15 sectors over the West Coast, East Coast, Gulf Coast, and Mississippi River area—echoed the challenges with the level of sector resources for MSRAM. For example, one district official stated that although Coast Guard headquarters has dedicated MSRAM staff, there are no full-time MSRAM analysts at the sector level. He added that each sector would need a dedicated person for MSRAM and risk analysis to bring MSRAM analysis into operational and tactical decision making.

Complexity of the MSRAM tool. Officials from 14 sectors (40 percent) reported that MSRAM use has been limited because data outputs require a substantial degree of analysis to use in decision making, or because the MSRAM tool itself is not easy to use. Some of the challenges raised by sectors that contribute to the complexity of the tool and interpreting its outputs included keeping abreast of yearly changes to the MSRAM tool and bridging knowledge gaps that occur when staff familiar with MSRAM rotate or leave the sector. In its MSRAM core document, the Coast Guard recognized that the frequent rotation of active duty personnel presents a risk to both the consistency of the MSRAM risk scoring efforts and the application of risk results.

Competing mission demands and resource constraints. Officials from 14 sectors (40 percent) reported that competing mission demands or resource constraints limited the use of MSRAM. Specifically, officials from 11 sectors reported that MSRAM’s usefulness was limited by the fact that it only considers risk in the PWCS mission, which is 1 of the Coast Guard’s 11 statutorily required missions. For example, a Great Lakes sector told us that while MSRAM identifies the risks in the sector, the sector is limited in its ability to move assets to address those security risks because the assets are also fulfilling other Coast Guard mission requirements, such as search and rescue. Additionally, officials from 6 sectors said that limited resources, such as boats or personnel, constrained their sectors’ ability to address the risks identified by MSRAM. For example, officials from 2 inland river sectors said that MSRAM identifies their security risks and demonstrates where they should patrol and plan for special events, but that they do not have the resources to carry out the plans. Further, officials from 1 of the inland river sectors added that their response boats are often busy escorting the Army Corps of Engineers or engaged in flood relief efforts. This leaves the work of security patrols to the local harbor patrol, which the officials said does not have the same capabilities, in terms of boats and weapons, as the Coast Guard.

Other challenges. Sector officials also identified other challenges with using MSRAM for informing decision making. Specifically, officials from 16 sectors (45 percent) said that MSRAM would be more useful if it was linked to other Coast Guard data systems, such as the Coast Guard’s inspections database, or if MSRAM was integrated into the sector command center. For example, officials from an east coast sector told us that they would like to see MSRAM linked to other databases in the sector command center, such as the Coast Guard’s vessel tracking system. Similarly, officials from a west coast sector said that integrating MSRAM into the Coast Guard’s inspections database would keep MSRAM continually updated and reflective of inspection results. Further, the command center has to consider other mission response needs, such as for pollution incidents or search and rescue, among others, and if MSRAM was integrated into the sector command center it could be used more in day-to-day operations. In addition, officials from 5 sectors noted that MSRAM does not capture dynamic risk, which limits its ability to inform daily decisions at the sector level. For instance, officials from a Gulf Coast sector said that they did not use MSRAM on a daily basis to allocate resources because daily fluctuations in vessel and barge risk are their greatest concern and this risk is not currently captured in MSRAM. The sectors that raised these issues believed that linking MSRAM into other data systems, integrating MSRAM into the command center, and having MSRAM account for dynamic risks could contribute to making its data more accurate, robust, and useful for decision making.

Coast Guard Has Taken Steps to Address Challenges, but Could Increase MSRAM’s Use by Expanding Training Opportunities

Coast Guard headquarters officials told us that they were aware of the challenges field-level MSRAM users were facing and have taken some steps to address them, but providing additional training could help integrate MSRAM throughout sector decision making. The Coast Guard’s current actions to address MSRAM user challenges include assessing the feasibility of adding additional risk analyst staff, increasing the data’s usability, developing decision-supporting modules, and providing training. These actions are described below.

Examining the feasibility of dedicated risk analysts. Presently, there is no dedicated risk analyst or MSRAM analyst position at the sector level, but headquarters officials told us in June 2011 that they are examining the feasibility of assigning additional port security specialists to the field and submitted a resource proposal for the additional staff. According to a senior Coast Guard budget official, given competing priorities and a constrained resource environment, it is unclear when or if this resource proposal will be funded.

Deploying MSRAM to sector command centers. To help make MSRAM more dynamic and increase its usability, the Coast Guard is piloting an Enterprise Geographic Information System (EGIS) display for sector command centers, which layers facility and vessel locations onto a satellite-based map and visually displays changing risk as vessels move into and out of ports. Officials from 7 sectors that participated in or were familiar with the initial EGIS test group reported that the functionality was very useful and had the potential to substantially increase MSRAM’s use for sector risk management efforts. In addition, headquarters officials told us in June 2011 that efforts were under way to integrate MSRAM into the Coast Guard’s inspections database, which would allow MSRAM to be continually updated and reflective of year-round facility and vessel inspection results.

Developing risk management modules. To assist with incorporating risk assessment information into decision making, in the fall of 2008, the Coast Guard began developing risk management modules within MSRAM that are able to provide specific types of analyses, such as comparing alternative security strategies. We asked officials from all 35 sectors their views on four modules—the Alternatives Evaluation Module, the Simplified Reporting Interface, the Daily Risk Profile, and the Risk Management Module. Sectors had mixed views on the utility of these modules. Specifically, officials from 14 sectors (40 percent) found the Alternatives Evaluation module very useful and cited such uses as evaluating Port Security Grant Program proposals and planning security for special events, and officials from 15 sectors (42 percent) found the Simplified Reporting Interface very useful for communicating risk information to port partners. However, with respect to the other two modules—the Daily Risk Profile and Risk Management Module—officials from 2 sectors (5 percent) found the Daily Risk Module very useful and officials from 3 sectors (8 percent) found the Risk Management Module very useful. For both modules, officials from 18 sectors (51 percent) reported that either they had not seen them or they were aware of the modules but did not have the time or training, among other reasons, to use them. Many of the modules are new and headquarters and some sector officials reported that they expected the modules would be more useful in the future as sectors gained familiarity with them through additional exposure and the annual MSRAM training.

Providing training. While the Coast Guard offers annual MSRAM training, officials from 25 sectors (71 percent) identified areas of the training for improvement, which the Coast Guard could do more to address. Specifically, officials from these sectors said that increasing the number of people who take MSRAM training, providing MSRAM training to command-level staff or senior management, and offering training on how to conduct risk analysis to inform decision making, among other things, would help integrate MSRAM throughout sector decision- making processes. Since MSRAM is a collateral duty, MSRAM training is not part of any Coast Guard personnel’s required training curriculum. However, Coast Guard guidance from May 2011 states that area, district, and sector commanders are responsible for ensuring that adequate numbers of appropriate personnel are trained in MSRAM. Only one sector did not, at the time of our interview, have at least one staff person trained in MSRAM. Officials from a Gulf Coast sector said that the training provided on the MSRAM tool itself is good, but the training does not teach the skills needed to make decisions in the field. Officials from a Great Lakes sector suggested that the Coast Guard develop an advanced course on how to use MSRAM to inform operational decisions. Officials from a southeastern sector added that the Coast Guard provides guidance on how to assess risks using MSRAM, but needs to provide more training on how to communicate MSRAM results and how those results can be used. In addition, a sector commanding officer who participated in one of our interviews told us that he was provided minimal training on MSRAM and wanted to understand more about how it can be used to support command-level decisions.

MSRAM has the capability of informing operational, tactical, and resource allocation decisions at all levels of a sector, but the Coast Guard has generally provided MSRAM training to a limited number of sector staff with specific MSRAM risk assessment responsibilities, such as port security specialists, rather than sector staff who may have command or management responsibilities where MSRAM may apply. Coast Guard headquarters officials said that this was because of limited resources to provide training for numerous sector personnel and variations in how MSRAM responsibilities are managed at different sectors. Standards for Internal Control in the Federal Government states that effective management of an organization’s workforce is essential to achieving results. Further, only when the right personnel for the job are on board and are provided the right training and tools, among other things, is operational success possible. To this end, management should ensure that training is aimed at developing and retaining employee skill levels to meet changing organizational needs. Coast Guard headquarters officials agree that providing MSRAM training to additional sector staff, particularly those with command and management responsibilities, would be valuable. Such training on how MSRAM can be used at all levels of command for risk-informed decision making—including how MSRAM can assist with the selection of different types of security measures to address areas of risk and the evaluation of their impacts—could further the Coast Guard’s efforts to implement its risk management framework and meet its goal to institutionalize MSRAM as the risk management tool for maritime security.

Coast Guard Measures Risk Reduction but Has Faced Challenges Using This Measure to Inform Decisions

Coast Guard Developed a Measure to Report Performance in Reducing Risk

The Coast Guard developed a performance measure and supporting model to measure and report its overall performance in reducing maritime security risk. This measure identifies the percentage reduction of maritime security risk, subject to Coast Guard influence, resulting from various Coast Guard actions. The Coast Guard considers this performance measure its key outcome measure for its PWCS mission. According to DHS’s Risk Management Fundamentals and the NIPP, it is crucial that a process of performance measurement be established to evaluate whether actions taken ultimately achieve the intended performance objective, such as reducing risk. This is important not only in evaluating program performance but also in holding the organization accountable for progress. We have also previously reported on the importance of developing outcome-based performance goals and measures as part of results management efforts. From fiscal years 2006 to 2010, the Coast Guard annually reported reducing from 15 to 31 percent of the maritime risk it is responsible for, in each year either meeting or exceeding its target. For fiscal years 2011 and 2012, the Coast Guard’s planned performance targets are to reduce more than 44 percent of the maritime security risk for which it is responsible.

To measure how its actions have reduced risk, the Coast Guard developed a model that uses a two-step approach. The first step is to estimate the total amount of terrorism risk that exists in the maritime domain, in the absence of any Coast Guard activities. This is referred to as raw risk, and this information comes primarily from MSRAM. second step relies on an elicitation process whereby Coast Guard subject matter experts estimate how various security activities and operations, maritime domain awareness programs, and regulatory structures— referred to by the Coast Guard as regimes—that the Coast Guard has implemented have reduced risk to U.S. ports and waterways. This step involves Coast Guard subject matter experts assessing the probability of these Coast Guard efforts failing to prevent a successful terrorist attack for 16 potential maritime terrorist attack scenarios.

Information also comes from DHS’s Risk Analysis Process for Informed Decision Making (RAPID) project, which is designed to provide strategic planning guidance and support resource allocation decisions at the DHS level.

According to DHS’s Risk Management Fundamentals, elicitations involve using structured questions to gather information from individuals with in-depth knowledge of specific areas or fields. missions, such as search and rescue, there is not a rich historical data set of maritime terrorism incidents that the Coast Guard can use to measure its actual performance. In other words, in the absence of an actual domestic maritime terrorism event, the Coast Guard uses internal subject matter experts to estimate risk reduction as a proxy measure of performance—an attempt to measure performance against a terrorism incident that did not occur.

Coast Guard Faces Challenges Using the Risk Reduction Performance Measure to Inform Decision Making

The Coast Guard’s efforts to develop an outcome measure to quantify the impact its actions have had on risk is a positive step. However, the use of the measure has been limited, and even with recent improvements, the Coast Guard faces challenges using this measure to inform decision making. Performance goals and measures are intended to provide Congress and agency management with information to systematically assess a program’s strengths, weaknesses, and performance. Thus, measures should provide information for management decision making. Coast Guard officials explained that the primary purpose of the risk reduction measure has been for external performance reporting, and to a more limited extent for informing strategic decision making and for conducting internal analysis of performance to identify areas for improvement. Specifically, officials said the measure has been used to compare risk across maritime terrorism scenarios and compare those results to other studies and analysis on maritime terrorism scenarios, which provided information on whether PWCS activities were appropriately balanced to address those risks. However, Coast Guard officials stated that over time, internal and external reviews identified limitations in the risk reduction measure, such as not allowing for comparisons of performance across sectors.

Recognizing these limitations, in 2010, the Coast Guard made improvements to the risk reduction model intended to enhance its utility for management decision making and to provide a more accurate measure of risk reduction. For example, the updated model includes information on the locations of Coast Guard assets and potential targets, which can be used to calculate the probability that Coast Guard assets will be able to intercept attacks. The Coast Guard also improved the elicitation techniques by which subject matter experts provided their estimates of Coast Guard risk reduction performance, and expanded the size and diversity of the subject matter experts involved in the elicitation According to Coast Guard officials, these improvements have process. made the measure and supporting model more useful for informing strategic decisions by allowing, for example, the ability to calculate risk reduction at the sector, district, area, and national levels and the risk reduction value of each element of the Coast Guard’s strategy. In other words, the updated model is able to show the risk reduction value of Coast Guard operational assets, such as small boats or helicopters, compared with regime activities, such as regulation enforcement. This information can help inform resource allocation decisions because it could identify which actions provide the greatest risk-reduction, according to these officials. The Coast Guard plans to use the updated model to measure its performance in reducing risk for the 2011 fiscal year.

According to the Coast Guard, in 2009 a total of 26 subject matter experts were used, mostly from headquarters. In 2010, a total of 46 subject matter experts were used coming from headquarters, areas, districts, sectors, and operational units. making. For example, given the inherent uncertainties in estimating risk reduction, it is unclear if a measure of risk reduction would provide meaningful performance information for tracking progress against goals and performance over time. According to our performance measurement criteria, to be able to assess progress toward the achievement of performance goals, the measures used must be reliable and valid. Reliability refers to the precision with which performance is measured, while validity is the extent to which the measure adequately represents actual performance. Therefore, the usefulness of agency performance information depends to a large degree on the reliability of performance data. We have also reported that decision makers must have assurance that the program data being used to measure performance are sufficiently reliable and valid if the data are to inform decision making. Although the Coast Guard has taken steps to improve the quality of the supporting model to provide a more accurate measure, estimating risk reduction is inherently uncertain and this measure is based on largely subjective judgments of Coast Guard personnel, and therefore the risk reduction results reported by the Coast Guard are not based on measurable or observable activities. As a result, it is difficult to independently verify or assess the validity or appropriateness of the judgments or to determine if this is an accurate measure of Coast Guard performance in the PWCS mission. However, Coast Guard officials told us that they believe these reported results provide a useful proxy measure of Coast Guard performance, and noted that this is one of several metrics the Coast Guard uses to assess performance in the PWCS mission.

According to DHS’s Risk Management Fundamentals, it is also important to be transparent about assumptions and key sources of uncertainty, so that decision makers are informed of the limitations of the risk information provided by the model. In its 2009 review of the risk reduction model, CREATE at the University of Southern California stated that it seemed likely that the model ignored important uncertainties and implied incorrectly high precision of risk estimates. Furthermore, OMB’s Updated Principles for Risk Analysis notes that because of the inherent uncertainties associated with estimates of risk, presentation of a single risk estimate may be misleading and provide a false sense of precision. OMB suggests that when a quantitative characterization of risk is provided, a range of plausible risk estimates should also be provided. From fiscal years 2006 to 2010, the Coast Guard reported the risk reduction measure as a specific risk reduction number rather than as a range of plausible risk reduction estimates. The Coast Guard official responsible for this measure told us this was because the previous risk reduction model was not capable of producing a range of plausible risk reduction estimates. The official noted that while the new risk reduction model—which will be used to report results for fiscal year 2011—is capable of producing a range of estimated risk reduction, the Coast Guard will continue to report the risk reduction measure as a single number because the DHS data system for performance reporting does not accept ranges—only numerical values. However, the official added that there is value in reporting a range of risk reduction and officials are considering a transition to a range of estimated reduction for the PWCS mission in future years. One alternative could be to report the percentage of risk reduced as a single number, but having an explanatory note indicating the range of plausible risk reduction estimates. Using a risk reduction measure that more accurately reflects performance effectiveness can give Coast Guard leaders and Congress a better sense of progress toward goals, which can support efforts to identify areas for improvement.

DHS officials have also raised some questions about the risk reduction measure. Recently, DHS determined that the Coast Guard’s risk reduction measure was not appropriate for inclusion as a DHS strategic performance measure and has designated it as a management measure. According to DHS, a strategic measure is designed to communicate achievement of strategic goals and objectives and be readily understandable to the public, and a management measure is designed to gauge program results and tie to resource requests and be used to support achievement of strategic goals. According to a senior DHS official, in 2010, DHS leadership reviewed all existing department measures and made decisions about which measures they believed were clearly tied to the DHS Quadrennial Homeland Security Review missions and were easily understandable by the public. This official noted that based on this review, DHS leadership did not feel the risk reduction measure and its methodology would be easily understandable by the public and therefore did not designate the measure as a strategic measure. As a result, the risk reduction measure will not be included in DHS’s annual performance plan, formally published with the Annual Performance Report, because this report only includes the smaller set of strategic measures. However, this official noted that the risk reduction measure is important as one piece of information to manage risk and is considered to be part of the full suite of DHS performance measures, and will continue to be published in the Coast Guard’s strategic context that is submitted with DHS’s Annual Performance Report.

Conclusions

The Coast Guard has invested substantial effort incorporating risk management principles into its security priorities and investments, and continues to proactively strengthen its assessment, management, and evaluation practices. As a result, the Coast Guard’s risk assessments and risk model are generally sound and in alignment with DHS standards. However, there are some additional actions that the Coast Guard could take to further its risk management approach by facilitating a wider use of risk information and making the results more valuable to the users. For example, since risk management is a tool for informing policymakers’ decisions about assessing risks, allocating resources, and taking actions under conditions of uncertainty, the Coast Guard could better document and communicate the uncertainty or confidence levels of its risk assessment results, including any implications that the uncertainty may have for decision makers. This added information would allow Coast Guard decision makers to prioritize strategies, tactics, and long-term investments with greater insight about the range of likely results and associated trade-offs with each decision. Additional information would also allow external reviewers of the risk model to reach the most appropriate conclusions or provide the most useful improvement recommendations through periodic reviews. The Coast Guard could also enhance the risk-informed prioritization of its field-level strategies, operations, and tactics by ensuring that risk management training is expanded to multiple levels of Coast Guard decision makers at the sector level, including command-level personnel. Expanding training on how MSRAM could be used at all levels of command for risk-informed decision making—including how MSRAM can assist with the selection of different types of security measures and the evaluation of their impacts—would further the Coast Guard’s efforts to implement its risk management framework and meet its goal of institutionalizing MSRAM as the risk management tool for maritime security. Finally, accurately representing performance results is important and the Coast Guard could more accurately convey its risk reduction performance measure by reporting risk reduction results as a range rather than a point estimate. Presenting risk reduction as a single number without a corresponding range of uncertainty could hamper Coast Guard efforts to identify areas for improvement. Taking these steps would make the Coast Guard’s risk management approach even stronger.

Recommendations for Executive Action

To help the Coast Guard strengthen MSRAM and better align it with NIPP risk management guidance, as well as facilitate the increased use of MSRAM across the agency, we recommend that the Commandant of the Coast Guard take the following three actions: (1) Provide more thorough documentation related to key assumptions and sources of uncertainty within MSRAM and inform users of any implications for interpreting the results from the model. (2) Make MSRAM available to appropriate parties for additional external peer review. (3) Provide additional training for sector command staff and others involved in sector management and operations on how MSRAM can be used as a risk management tool to inform sector-level decision making.

To improve the accuracy of the risk reduction measure for internal and external decision-making, we recommend that the Commandant of the Coast Guard take action to report the results of the risk reduction measure as a range rather than a point estimate.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS and the Coast Guard on October 17, 2011, for review and comment. DHS provided written comments, which are reprinted in appendix II. DHS and the Coast Guard concurred with the findings and recommendations in the report, and stated that the Coast Guard is taking actions to implement our recommendations.

The Coast Guard concurred with our first recommendation that it provide more thorough documentation related to key assumptions and sources of uncertainty within MSRAM. Specifically, the Coast Guard stated that the documentation of uncertainty is part of the ongoing MSRAM VV&A process, and that the Coast Guard will continue to work with the DHS Office of Risk Management and Analysis in developing a feasible and deployable model that will benefit field-level security operations. These actions should improve the Coast Guard’s ability to document and inform MSRAM users of any implications for interpreting results from the model, thereby addressing the intent of our recommendation.

Regarding the second recommendation that the Coast Guard make MSRAM available to appropriate parties for additional external peer review, the Coast Guard concurred. The Coast Guard stated that external peer review is part of the ongoing MSRAM VV&A process, and that additional external peer review will be part of an independent verification and validation of MSRAM expected to be completed in the fall of 2012. Such actions should address the intent of the recommendation.

Regarding the third recommendation that the Coast Guard provide additional training for sector command staff and others involved in sector management on how MSRAM can be used as a risk management tool, the Coast Guard concurred. Specifically, the Coast Guard stated that MSRAM is part of the Coast Guard’s contingency planning course, and the Coast Guard will explore other opportunities to provide risk training to sector command staff, including online and webinar training opportunities. Such actions, once implemented, should address the intent of the recommendation.

Finally, the Coast Guard also concurred with the fourth recommendation to take action to report the results of the risk reduction measure as a range rather than a point estimate. The Coast Guard stated that it is currently limited by the DHS data reporting system with regard to the format of presenting performance targets and results, but noted that it is currently working with DHS to determine options for reporting risk as a range. Such action, when fully implemented, should address the intent of the recommendation.

DHS and the Coast Guard also provided us with technical comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Homeland Security, appropriate congressional committees, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any further questions about this report, please contact me at (202) 512-9610 or caldwells@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors are listed in appendix III.

Appendix I: Risk Management Framework

To provide guidance to agency decision makers, we developed a risk management framework which is intended to be a starting point for applying risk-informed principles. Our risk management framework, shown in figure 2, entails a continuous process of managing risk through a series of actions, including setting strategic goals and objectives, assessing risk, evaluating alternatives, selecting initiatives to undertake, and implementing and monitoring those initiatives.

Setting strategic goals, objectives, and constraints is a key first step in applying risk management principles and helps to ensure that management decisions are focused on achieving a purpose. Risk assessment, an important element of a risk-informed approach, helps decision makers identify and evaluate potential risks so that countermeasures can be designed and implemented to prevent or mitigate the effects of the risks. Risk assessment is a qualitative determination, quantitative determination, or both of the likelihood of an adverse event occurring and the severity, or impact, of its consequences. Risk assessment in a homeland security application involves assessing three key components—threat, vulnerability, and consequence. A threat assessment is the identification and evaluation of adverse events that can harm or damage an asset. A vulnerability assessment identifies weaknesses in physical structures, personal protection systems, processes, or other areas that may be exploited. A consequence assessment is the process of identifying or evaluating the potential or actual effects of an event, incident, or occurrence. Information from these three assessments contributes to an overall risk assessment that characterizes risks, which can provide input for evaluating alternatives and prioritizing security initiatives. The risk assessment element in the overall risk management cycle informs each of the remaining steps of the cycle. Alternatives evaluation addresses the evaluation of risk reduction methods by consideration of countermeasures or countermeasure systems and the costs and benefits associated with them. Management selection addresses such issues as determining where resources and investments will be made, the sources and types of resources needed, and where those resources would be targeted. The next phase in the framework involves the implementation of the selected countermeasures. Following implementation, monitoring is essential to help ensure that the entire risk management process remains current and relevant and reflects changes in the effectiveness of the alternative actions and the risk environment in which it operates. Program evaluation is an important tool for assessing the efficiency and effectiveness of the program. As part of monitoring, consultation with external subject area experts can provide a current perspective and an independent review in the formulation and evaluation of the program.

The National Infrastructure Protection Plan (NIPP), originally issued by the Department of Homeland Security (DHS) in 2006 and updated in 2009, includes a risk analysis and management framework, which, for the most part, mirrors our risk management framework. This framework includes six steps—set goals and objectives; identify assets, systems, and networks; assess risks; prioritize; implement programs; and measure effectiveness. The NIPP is DHS’s base plan that guides how DHS and other relevant stakeholders should use risk management principles to prioritize protection activities. In 2009, DHS updated the NIPP to, among other things, increase its emphasis on risk management, including an expanded discussion of risk management methodologies and discussion of a common risk assessment approach that provided core criteria for these analyses. Beyond the NIPP, DHS has issued additional risk management guidance and directives. For example, in January 2009 DHS published its Integrated Risk Management Framework, which, among other things, calls for DHS to use risk assessments to inform decision making. In April 2011, DHS issued its Risk Management Fundamentals, which establishes specific doctrine and guidance for risk management across DHS.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Dawn Hoff, Assistant Director and Adam Hoffman, Analyst-in-Charge, managed this assignment. Chuck Bausell, Charlotte Gamble, and Grant Sutton made significant contributions to this report. Colleen McEnearney provided assistance with interviews and data analysis. Michele Fejfar assisted with design, methodology, and data analysis. Jessica Orr provided assistance with report development, and Geoff Hamilton provided legal assistance.

Related GAO Products

Port Security Grant Program: Risk Model, Grant Management, and Effectiveness Measures Could Be Strengthened. GAO-12-47. Washington, D.C.: November 17, 2011.

Maritime Security: Progress Made but Further Actions Needed to Secure the Maritime Energy Supply. GAO-11-883T. Washington, D.C.: August 24, 2011.

Maritime Security: DHS Progress and Challenges in Key Areas of Port Security. GAO-10-940T. Washington, D.C.: July 21, 2010.

Maritime Security: Varied Actions Taken to Enhance Cruise Ship Security, but Some Concerns Remain. GAO-10-400. Washington, D.C.: April 9, 2010.

Critical Infrastructure Protection: Update to National Infrastructure Protection Plan Includes Increased Emphasis on Risk Management and Resilience. GAO-10-296. Washington, D.C.: March 5, 2010.

Transportation Security: Comprehensive Risk Assessments and Stronger Internal Controls Needed to Help Inform TSA Resource Allocation. GAO-09-492. Washington, D.C.: March 27, 2009.

Department of Homeland Security: Progress Report on Implementation of Mission and Management Functions. GAO-07-454. Washington, D.C.: August 17, 2007.

Risk Management: Further Refinements Needed to Assess Risks and Prioritize Protective Measures at Ports and Other Critical Infrastructure. GAO-06-91. Washington, D.C.: December 15, 2005.

Managing for Results: Enhancing Agency Use of Performance Information for Management Decision Making. GAO-05-927. Washington, D.C.: September 9, 2005.

Homeland Security: Summary of Challenges Faced in Targeting Oceangoing Cargo Containers for Inspection. GAO-04-557T. Washington D.C.: March 31, 2004.

Managing for Results: Challenges Agencies Face in Producing Credible Performance Information. GAO/GGD-00-52. Washington, D.C.: February 4, 2000.

The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1998.