Coast Guard: Security Risk Model Meets DHS Criteria, but More Training Could Enhance Its Use for Managing Programs and Operations
Published: 2011-11-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since the terrorist attacks of September 11, 2001, the nation's ports and waterways have been viewed as potential targets of attack. The Department of Homeland Security (DHS) has called for using risk-informed approaches to prioritize its investments, and for developing plans and allocating resources that balance security and the flow of commerce. The U.S. Coast Guard--a DHS component and the lead federal agency responsible for maritime security--has used its Maritime Security Risk Analysis Model (MSRAM) as its primary approach for assessing and managing security risks. GAO was asked to examine (1) the extent to which the Coast Guard's risk assessment approach aligns with DHS risk assessment criteria, (2) the extent to which the Coast Guard has used MSRAM to inform maritime security risk decisions, and (3) how the Coast Guard has measured the impact of its maritime security programs on risk in U.S. ports and waterways. GAO analyzed MSRAM's risk assessment methodology and interviewed Coast Guard officials about risk assessment and MSRAM's use across the agency.
section_title: What GAO Found
paragraphs: MSRAM generally aligns with DHS risk assessment criteria, but additional documentation on key aspects of the model could benefit users of the results. MSRAM generally meets DHS criteria for being complete, reproducible, documented, and defensible. Further, the Coast Guard has taken actions to improve the quality of MSRAM data and to make them more complete and reproducible, including providing training and tools for staff entering data into the model. However, the Coast Guard has not documented and communicated the implications that MSRAM's key assumptions and other sources of uncertainty have on MSRAM's risk results. For example, to assess risk in MSRAM, Coast Guard analysts make judgments regarding such factors as the probability of an attack and the economic and environmental consequences of an attack. These multiple judgments are inherently subjective and constitute sources of uncertainty that have implications that should be documented and communicated to decision makers. Without this documentation, decision makers and external MSRAM reviewers may not have a complete understanding of the uses and limitations of MSRAM data. In addition, greater transparency and documentation of uncertainty and assumptions in MSRAM's risk estimates could also facilitate periodic peer reviews of the model--a best practice in risk management. MSRAM is the Coast Guard's primary tool for managing maritime security risk, but resource and training challenges hinder use of the tool by Coast Guard field operational units, known as sectors. At the national level, MSRAM supports Coast Guard strategic planning efforts, which is consistent with the agency's intent for MSRAM. At the sector level, MSRAM has informed a variety of decisions, but its use has been limited by lack of staff time, the tool's complexity, and competing mission demands, among other things. The Coast Guard has taken actions to address these challenges, but providing additional training on how MSRAM can be used at all levels of sector decision making could further the Coast Guard's risk management efforts. MSRAM is capable of informing operational, tactical, and resource allocation decisions, but the Coast Guard has generally provided MSRAM training only to a small number of sector staff who may not have insight into all levels of sector decision making. The Coast Guard developed an outcome measure to report its performance in reducing maritime risk, but has faced challenges using this measure to inform decisions. Outcome measures describe the intended result of carrying out a program or activity. The measure is partly based on Coast Guard subject matter experts' estimates of the percentage reduction of maritime security risk subject to Coast Guard influence resulting from Coast Guard actions. The Coast Guard has improved the measure to make it more valid and reliable and believes it is a useful proxy measure of performance, noting that developing outcome measures is challenging because of limited historical data on maritime terrorist attacks. However, given the uncertainties in estimating risk reduction, it is unclear if the measure would provide meaningful performance information with which to track progress over time. In addition, the Coast Guard reports the risk reduction measure as a specific estimate rather than as a range of plausible estimates, which is inconsistent with risk analysis criteria. Reporting and using outcome measures that more accurately reflect mission effectiveness can give Coast Guard leaders and Congress a better sense of progress toward goals.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Coast Guard provide more thorough documentation on MSRAM's assumptions and other sources of uncertainty, make MSRAM available for peer review, implement additional MSRAM training, and report the results of its risk reduction performance measure in a manner consistent with risk analysis criteria. The Coast Guard agreed with these recommendations.

section_title: Letter
paragraphs: Since the terrorist attacks of September 11, 2001, the nation’s ports have  been viewed as potential targets of attack for many reasons. Ports,  waterways, and vessels are part of an economic engine handling more  than $700 billion in merchandise annually, according to the Department of  Homeland Security (DHS), and an attack on this system could have a  widespread impact on global shipping, international trade, and the U.S.  economy. The U.S. Coast Guard—a component of DHS—is the lead  federal agency for maritime security, which includes the protection of U.S.  ports, coasts, and inland waterways as part of its Ports, Waterways, and  Coastal Security (PWCS) mission. This mission involves protecting the  maritime domain and marine transportation system, including preventing  terrorist attacks, and responding to and recovering from attacks that do  occur. In addition to its PWCS mission, the Coast Guard has 10 other  statutory missions.
Since it is not practical or economically feasible to protect all assets  against every possible terrorist risk, DHS has called for using risk- informed approaches to prioritize its investments and for developing plans  and allocating resources that balance security and the flow of commerce.  Risk management is a tool for informing policymakers’ decisions about  assessing risks, allocating resources, and taking actions under conditions  of uncertainty. A risk management approach entails a continuous process  of managing risk through a series of actions, including setting strategic  goals and objectives, assessing risk, evaluating alternatives, selecting  initiatives to undertake, and implementing and monitoring those initiatives.  DHS detailed this approach in its National Infrastructure Protection Plan  (NIPP), which it issued in June 2006 and updated in 2009. The 2009 update of the NIPP increased the plan’s emphasis on risk management,  including providing the core criteria of a risk assessment approach.
The Coast Guard’s primary approach to assessing and managing security  risks has been embodied in its Maritime Security Risk Analysis Model  (MSRAM). Since its development and implementation in 2005, MSRAM  has provided the Coast Guard with a standardized way of assessing risk  to maritime infrastructure, referred to in MSRAM as targets, which can  include chemical facilities, oil refineries, hazardous cargo vessels,  passenger ferries, and cruise ship terminals, to name a few. MSRAM is  designed to allow comparison between different targets at the local,  regional, and national levels with the goal of reducing risk by prioritizing  security activities and resources. MSRAM calculates the risk of terrorist  attack based on scenarios—a combination of target and attack mode—in  terms of threats, vulnerabilities, and consequences to more than 28,000  maritime targets. For example, a MSRAM scenario related to cruise ships  could include a boat bomb or an attack by a hijacked vessel.
Since 2004, we have examined Coast Guard efforts to implement a risk  management framework, noting how the Coast Guard’s risk management  and risk assessment efforts have developed and evolved, as well as how  the Coast Guard has made progress in assessing maritime security risks  using MSRAM. For example, in 2005, we reported that by developing  MSRAM, the Coast Guard had begun to address the limitations of its  previous port security risk model.Guard has strengthened risk management through the development and  use of MSRAM to help prioritize limited port security resources, identify  capabilities needed to combat future threats, and identify the highest-risk   In 2010, we reported that the Coast  scenarios and targets in the maritime domain. We also reported in 2010  that the Coast Guard was assessing risk to cruise ships and facilities in  accordance with DHS risk assessment guidance and that the Coast  Guard was using MSRAM to help concentrate maritime security activities  where relative risk is believed to be greatest.environment combined with the Coast Guard’s multiple missions, it is  critically important for the Coast Guard to make the most effective use of  its limited resources and to ensure that all levels of the Coast Guard are  equipped to make risk-informed decisions regarding maritime security.
In light of the tight fiscal  You asked us to examine the Coast Guard’s progress in using MSRAM to  assess and manage maritime security risk and to assess its progress  implementing DHS’s risk management framework—specifically, how the  Coast Guard is establishing security priorities based on risk,  implementing protective programs and strategies, and measuring the  effectiveness of its actions. This report addresses the following questions:
To what extent does the Coast Guard’s risk assessment approach  align with DHS risk assessment criteria, and what challenges, if any,  exist in this effort?
To what extent has the Coast Guard used MSRAM to inform maritime  security risk decisions, and what challenges, if any, exist in this effort?
How has the Coast Guard measured the impact of its maritime  security programs on risk in U.S. ports and waterways, and what  challenges, if any, exist in this effort?
To address our first objective, we focused on MSRAM, which is the Coast  Guard’s primary model for assessing maritime security risk. We  compared MSRAM’s risk assessment methodology and processes to  relevant criteria, including the risk assessment component of the 2009  NIPP and our related reports on risk management, such as our 2005  report examining Coast Guard risk management efforts. We interviewed  Coast Guard headquarters officials to discuss their national-level  perspectives on the overall management and implementation of the  MSRAM risk model and we interviewed officials from the Coast Guard’s  Intelligence Coordination Center to discuss how intelligence information is  incorporated into MSRAM. In addition, we interviewed officials from all  35 Coast Guard sectors to obtain their views on MSRAM. We asked  these officials to provide information on the MSRAM risk assessment  process, MSRAM training, and processes and procedures for ensuring  MSRAM data integrity. The interview was a mix of specific questions  using a rating scale and questions asking for open-ended or narrative  responses. During these interviews, we interviewed staff responsible for  collecting and updating MSRAM data, as well as management-level  officials in the sectors’ response, prevention, planning, and command  units. To minimize any inconsistencies or errors in the information we  collected, our subject matter experts developed the interview questions in  collaboration with a social science survey specialist and we pretested the  interview with officials from two Coast Guard sectors. We provided an  advance copy of the interview questions to each of the sectors to allow  time to prepare responses, conduct preliminary research, and identify  appropriate points of contact. We also validated selected interview  response information by corroborating it with other sources, such as  MSRAM documentation provided by the Coast Guard. We conducted  these interviews from May 2011 through August 2011. We also  conducted interviews with officials from three of the nine Coast Guard  districts to obtain their perspectives on MSRAM. These districts  encompass 15 sectors over the West Coast, East Coast, Gulf Coast, and  Mississippi River area. Since we selected a nonprobability sample of  districts, the information obtained from these interviews cannot be  generalized to all districts but provides us with information on how officials  from these selected districts view MSRAM. In addition, we reviewed  external studies on MSRAM, including a MSRAM verification and  validation report and a report by the National Center for Risk and  Economic Analysis of Terrorism Events (CREATE) at the University of  Southern California, and we met with the author of the MSRAM  verification and validation report to discuss the report’s findings and  recommendations. We also reviewed the National Research Council of  the National Academies study on DHS risk models and spoke with  individuals involved in conducting that study. Although this study did not  specifically review MSRAM, it provided broad information on risk analysis  and modeling applicable to MSRAM. We reviewed the methodologies of  these studies and found them sufficiently reliable for the purposes of our  report. In addition, we met with risk management experts from DHS’s  Office of Risk Management and Analysis to obtain their views on risk  analysis. We also met with an external risk management expert familiar  with MSRAM and Coast Guard risk management efforts to gain additional  perspective on MSRAM. While our review focused on MSRAM, we also  obtained information regarding other Coast Guard risk management  models that are under development to see how these models are  expected to align with MSRAM.
To address our second objective, we also focused on MSRAM because it  is the Coast Guard’s primary maritime security risk management tool. We  reviewed Coast Guard documents describing MSRAM’s current and  intended uses at headquarters and in the field. We also relied on the  interviews with officials from the 35 Coast Guard sectors and the 3 Coast  Guard districts as previously described. During these interviews, we  obtained information on how sectors and districts use MSRAM at the local  level to guide tactical, operational, and strategic security efforts and  strengths and limitations of MSRAM for these purposes. Based on the  information provided by these interviews, we compared sectors’ reported  uses of MSRAM for informing risk-based decision making with Coast  Guard documentation on the intended uses of MSRAM, including  Commandant Instructions and internal Coast Guard risk management  guidance documents. We verified the intended uses of MSRAM with  officials from Coast Guard headquarters. At Coast Guard headquarters,  we interviewed personnel from Coast Guard offices that use MSRAM  information to support decision making, as well as officials responsible for  developing risk assessment training standards within the Coast Guard.
To address our third objective, we focused on the Coast Guard’s risk  reduction performance measure and its supporting model. This measure  is the Coast Guard’s primary method for measuring and reporting its  overall performance in reducing risk in the maritime domain. We reviewed  Coast Guard documentation on the risk reduction measure and  supporting model, as well as relevant criteria, including our criteria for  performance measurement, the NIPP framework, and the Office of  Management and Budget (OMB) Updated Principles for Risk Analysis.  Additionally, we interviewed Coast Guard officials to discuss how the risk  reduction measure and supporting model are used as well as recent and  planned improvements. We also discussed the Coast Guard’s risk  reduction measure with a senior DHS official responsible for reviewing  department and component-level performance measures.
We conducted this performance audit from October 2010 through  November 2011 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Risk Management
paragraphs: In recent years, we, Congress, the 9/11 Commission, and others have  recommended that federal agencies with homeland security  responsibilities utilize a risk management approach to help ensure that  finite resources are dedicated to assets or activities considered to have  the highest security priority. The purpose of risk management is not to  eliminate all risks, as that is an impossible task. Rather, given limited  resources, risk management is a structured means of making informed  trade-offs and choices about how to use available resources effectively  and monitoring the effect of those choices. Thus, risk management is a  continuous process that includes the assessment of threats,  vulnerabilities, and consequences to determine what actions should be  taken to reduce or eliminate one or more of these elements of risk.
To provide guidance to agency decision makers, we developed a risk  management framework, which is intended to be a starting point for  applying risk-informed principles. Our risk management framework entails  a continuous process of managing risk through a series of actions,  including setting strategic goals and objectives, assessing risk, evaluating  alternatives, selecting initiatives to undertake, and implementing and  monitoring those initiatives. Additional information on risk management,  including our risk management framework, can be found in appendix I.
DHS is required by statute to utilize risk management principles with  respect to various DHS functions. With regard to the Coast Guard,  federal statutes call for the Coast Guard to use risk management in  specific aspects of its homeland security efforts. The Maritime  Transportation Security Act of 2002 (MTSA), for example, calls for the  Coast Guard and other port security stakeholders, through implementing  regulations, to carry out certain risk-based tasks, including assessing  risks and developing security plans for ports, facilities, and vessels. In  addition, the Coast Guard Authorization Act of 2010 requires, for  example, the Coast Guard to (1) develop and utilize a national standard  and formula for prioritizing and addressing assessed security risks at U.S.  ports and facilities, such as MSRAM; (2) require Area Maritime Security  Committees to use this standard to regularly evaluate each port’s  assessed risk and prioritize how to mitigate the most significant risks; and  (3) make MSRAM available, in an unclassified version, on a limited basis  to regulated vessels and facilities to conduct risk assessments of their  own facilities and vessels.
section_title: Coast Guard Security Risk Models
paragraphs: From 2001 to 2006, the Coast Guard assessed maritime security risk  using the Port Security Risk Assessment Tool (PSRAT), which was  quickly developed and fielded after the terrorist attacks of September 11,  2001. PSRAT served as a rudimentary risk calculator that ranked  maritime critical infrastructure and key resources (MCIKR) with respect to  the consequences of a terrorist attack and evaluated vessels and facilities  that posed a high risk of a transportation security incident. While PSRAT  provided a relative risk of targets within a port region, it could not  compare and prioritize relative risks of various infrastructures across  ports, among other limitations.
Recognizing the shortcomings of PSRAT that had been identified by the  Coast Guard and us, in 2005 the Coast Guard developed and  implemented MSRAM to provide a more robust and defensible terrorism  risk analysis process. MSRAM is a risk-based decision support tool  designed to help the Coast Guard assess and manage maritime security  risks throughout the Coast Guard’s area of responsibility. Coast Guard  units throughout the country use this tool to assess security risks to over  28,000 key maritime infrastructure assets—also known as targets—such  as chemical facilities, passenger terminals, and bridges, as well as  vessels such as cruise ships, ferries, and vessels carrying hazardous  cargoes, among other things. Unlike PSRAT, MSRAM is designed to  capture the security risks facing different types of targets, allowing  comparison between different targets and geographic areas at the local,  regional, and national levels. MSRAM’s risk assessment methodology  assesses the risk of a terrorist attack based on different scenarios; that is,  it combines potential targets with different attack modes for each  target/attack mode combination (see table 1). MSRAM automatically  determines which attack modes are required to be assessed for each  target type, though local MSRAM analysts have the ability to evaluate  additional optional attack modes against any target. For each  target/attack mode combination, MSRAM can provide different risk  results, such as the inherent risk of a target and the amount of risk  mitigated by Coast Guard security efforts.
MSRAM calculates risk using the following risk equation: Risk = Threat x  Vulnerability x Consequence. Numerical values representing Coast  Guard’s assessment of threat (or relative likelihood of attack),  vulnerability should an attack occur, and consequences of a successful  attack are combined to yield a risk score for each maritime target. The  model calculates risk using threat judgments provided by the Coast  Guard Intelligence Coordination Center (ICC), and vulnerability and  consequence judgments provided by MSRAM users at the sector level— typically Coast Guard port security specialists—which are reviewed at the  district, area, and headquarters levels. The risk equation variables are as  follows:  Threat represents the relative likelihood of an attempted attack on a  target. The ICC provides threat probabilities to MSRAM, based upon  judgments regarding specific intent, capability, and geographic preference  of terrorist organizations to deliver an attack on a specific type of maritime  target class—for example, a boat bomb attack on a ferry terminal. To  make these judgments, ICC officials use intelligence reports generated  throughout the broader intelligence community to make qualitative  determinations about certain terrorist organizations and the threat they  pose to the maritime domain. At the sector level, Coast Guard MSRAM  users do not input threat probabilities and are required to use the threat  probabilities provided by the ICC. This approach is intended to ensure  that threat information is consistently applied across ports.
Vulnerability represents the probability of a successful attack given an  attempt. MSRAM users at the sector level assess the vulnerability of  targets within their respective areas of responsibility. Table 2 shows the  factors included in the MSRAM vulnerability assessment.
Consequence represents the projected overall impact of a successful  attack on a given target or asset. Similar to vulnerability assessments,  MSRAM users at the sector level assess the consequences of a  successful attack on targets within their respective area of responsibility.  Table 3 shows the factors included in the MSRAM consequence  assessment.
In addition to the consequence factors listed in table 3, sector MSRAM  users also assess the response capabilities of the Coast Guard, port  stakeholders, and other governmental agencies and their ability to  mitigate death/injury, primary economic, and environmental  consequences of a successful attack. Because there is a broad array of  target types operating in the maritime domain that can result in different  types of impacts if successfully attacked, MSRAM uses an approach for  drawing equivalencies between the different types of impacts. This  approach was based on establishing a common unit of measure, called a  consequence point. One consequence point represents $1 million of  equivalent loss to the American public.
To support MSRAM development and risk analysis at the headquarters  level, the Coast Guard has provided MSRAM-dedicated staff and  resources. According to the Coast Guard, resources for MSRAM or port  security risk analysis are not from a specific budget line item. From fiscal  year 2006 to fiscal year 2011, the Coast Guard reported assigning from  two to five staff (full-time equivalents) and from $0.6 million to $1.0 million  annually to support MSRAM at headquarters. There are no MSRAM- dedicated staff at the area, district, and sector levels; rather, MSRAM  assessment and analysis is generally conducted by port security  specialists, who have other responsibilities. The port security specialist  typically has responsibility for numerous activities, including the Port  Security Grant Program, Area Maritime Security Committees, and Area  Maritime Security Training Exercise Program, among others.
section_title: DHS Risk Management Criteria
paragraphs: The NIPP is DHS’s primary guidance document for conducting risk  assessments and includes core criteria that identify the characteristics and  information needed to produce quality risk assessment results. The NIPP’s  basic analytical principles state that risk assessments should be complete,  reproducible, documented, and defensible, as defined in table 4.
section_title: MSRAM Risk Assessments Generally Align with DHS Criteria, but Challenges Remain
paragraphs: MSRAM generally aligns with DHS’s criteria for a complete and  reproducible risk assessment, but some challenges remain, such as the  limited time for Coast Guard personnel to complete assessments.  MSRAM also generally aligns with the NIPP criteria for a documented and  defensible risk assessment, but the Coast Guard could improve its  documentation of the model’s assumptions and other sources of  uncertainty, such as the subjective judgments made by Coast Guard  analysts about vulnerabilities and consequences, and how these  assumptions and other sources of uncertainty affect MSRAM’s results. In  addition to providing decision makers with an understanding of how to  interpret any uncertainty in MSRAM’s risk estimates, greater transparency  and documentation could facilitate periodic peer reviews of the model—a  best practice in risk management.
subsections: section_title: MSRAM Generally Aligns with NIPP Criteria for a Complete and Reproducible Risk Assessment
paragraphs: MSRAM generally aligns with NIPP criteria for a complete risk  assessment. In accordance with NIPP criteria for a complete risk  assessment, MSRAM assesses risk using three main variables— consequence, vulnerability, and threat. MSRAM’s risk assessment  methodology also follows the NIPP criteria for factors that should be  assessed in each of the three risk variables. Specifically, for threat,  MSRAM generally follows the NIPP criteria by identifying attack methods  that may be employed and by considering the adversary’s intent and  capability to attack a target. MSRAM generally follows the vulnerability  assessment criteria by estimating the likelihood of an adversary’s success  for each attack scenario and describing the protective measures in place,  and MSRAM generally follows the consequence assessment criteria by  estimating economic loss in dollars, estimating fatalities, and describing  psychological impacts, among other things.
MSRAM’s risk assessment methodology also generally aligns with the  NIPP criteria for a reproducible risk assessment. To be reproducible, the  methodology must produce comparable, repeatable results and minimize  the number and impact of subjective judgments, among other things.  Although Coast Guard officials acknowledge that MSRAM risk data are  inherently subjective, the MSRAM model and data collection processes  include features designed to produce comparable, repeatable results  across sectors. For instance, the Coast Guard prepopulates threat data  into MSRAM from the Coast Guard’s ICC. This allows for nationally vetted  threat scores that do not rely on multiple subjective local judgments. DHS,  in its 2010 Transportation Systems Sector-Specific Plan, stated that  MSRAM produces comparable, repeatable results.
The Coast Guard has taken numerous actions that contribute to MSRAM  being a complete and reproducible risk assessment model. To improve the  quality and accuracy of MSRAM data and reduce the amount of subjectivity  in the MSRAM process, the Coast Guard conducts an annual review and  validation of MSRAM data produced at each sector; provides MSRAM  users with tools, calculators, and benchmarks to assist in calculating  consequence and vulnerability; and provides training to sectors on how to  enter data into MSRAM. Specific actions are detailed below.
Annual validation and review. The Coast Guard uses a multilevel  annual validation and review process, which helps to ensure that MSRAM  risk data are comparable and repeatable across sectors. According to a  2010 review of MSRAM, conducting a thorough review process across  sectors is especially important if the data are to be used for national-level  decision making. This process includes sector, district, area, and  headquarters officials and aims to normalize MSRAM data by establishing  national averages of risk scores for attack modes and targets and by  identifying outliers. The annual MSRAM validation and review process  begins with sectors completing vulnerability and consequence  assessments for targets within their areas of responsibility. Once the  sector Captain of the Port validates the assessments, the risk  assessment data are sent to district and area officials for review.  Following these reviews, Coast Guard headquarters officials combine  each sector’s data into a national classified dataset and perform a  statistical analysis of the data. The statistical analysis involves calculating  national averages for vulnerability, consequence, and response   When determining whether a sector’s risk score  capabilities risk scores.for a specific target is questionable or is an outlier, reviewers consider the  results of the statistical analysis as well as supporting comments or  rationale provided by sector officials.
According to the Coast Guard, for each outlier identified during the  national review process, sector officials reconsider the data point in  question and either change the inputs to reflect national averages or  provide additional justification for why the risk score for the target in  question should be outside of the national average. Headquarters officials  explained that they generally accept justification for data outliers and that  a goal of the review process is to spur discussions related to maritime risk  rather than forcing compliance with national data averages. For example,  officials from one sector told us that a small port in their sector is critical  for their state’s energy imports, and accordingly, the port infrastructure is  high risk on a national scale. The officials said that Coast Guard  headquarters officials have questioned the relatively high risk rankings of  the port’s infrastructure because they are statistical outliers, but have  deferred to the expertise of the sector regarding the risk scores.
Tools and calculators. Recognizing that sector port security specialists  who assess risk using MSRAM generally do not have expertise in all  aspects of assessing vulnerability and consequence, the Coast Guard  has regularly added new tools and calculators to MSRAM to improve the  quality, accuracy, and consistency of vulnerability and consequence  assessments. For example, MSRAM now includes a blast calculator that  allows users to more easily determine the death and injury consequences  of an explosion close to population centers. Coast Guard officials from 29  sectors (82 percent of sectors) cited a variety of challenges with  assessing vulnerability and consequence values in MSRAM, but officials  from 10 sectors said that it was becoming easier to do over time and  officials from 14 sectors said that the tools and calculators in MSRAM  have helped.
Benchmarks and recommended ranges. To limit inconsistencies  caused by different judgments by individual MSRAM users and to  minimize user subjectivity, the Coast Guard built into MSRAM a  suggested range of scores for each risk factor—including vulnerability,  consequence, and response capabilities—as well as averages, or  benchmarks, of scores for each factor. The benchmarks are based on  Coast Guard and expert evaluation of target classes and attack modes.  The benchmarks and recommended ranges are reviewed and updated  each year following the annual data revalidation cycle.
Training. The Coast Guard has also provided annual training for MSRAM  users, including beginning, intermediate, and advanced courses intended  to standardize the data entry process across the Coast Guard. Officials  from 34 sectors (97 percent) reported finding the training moderately to  very useful in terms of enhancing their ability to assess, understand, and  communicate the risks facing their sectors. In 2011, Coast Guard  headquarters also started providing live web-based training sessions on  various MSRAM issues, such as resolving national review comments, to  help sector staff gain familiarity with MSRAM’s features on an as-needed  basis. In addition to MSRAM training provided by headquarters, one  Coast Guard district official we spoke with had developed and provided  localized training to the sector-level port security specialists on assessing  the vulnerability of chemical facilities. The district official told us that  Coast Guard headquarters was interested in this local model for  delivering training and was planning to pilot a similar training program in a  different district.
MSRAM generally aligns with DHS’s criteria for a complete and  reproducible risk assessment, but challenges remain with the MSRAM  methodology and risk assessment process. The Coast Guard has  acknowledged these challenges and limitations and has actions underway  to address them and make MSRAM more complete and reproducible.  Coast Guard officials noted that some of these challenges are not unique to  MSRAM and are faced by others in the homeland security risk assessment  community. Specific challenges are detailed below.
Data subjectivity. While the Coast Guard has taken actions to minimize  the subjectivity of MSRAM data, officials acknowledged that assessing  threat, vulnerability, and consequence is inherently subjective. To assess  threat, the Coast Guard’s ICC quantifies judgments related to the intent  and capability of terrorist organizations to attack domestic maritime  infrastructure. However, there are limited national historic data for  domestic maritime attacks and thus intelligence officials must make a  number of subjective judgments and draw inferences from international  maritime attacks. Further, GAO has previously reported on the inherently difficult nature of assessing the capability and intent of terrorist groups.Vulnerability and consequence assessments in MSRAM are also  inherently subjective. For example, officials from 20 sectors we  interviewed said that even with training, tools, and calculators, assessing  consequences can be challenging and that it often involved subjectivity  and uncertainty. Officials noted that assessing economic impacts—both  primary and secondary—was particularly challenging because it required  some level of expertise in economics—such as supply chains and  industry recoverability—which port security specialists said is often  beyond their skills and training. The input for secondary economic  impacts can have a substantial effect on how MSRAM’s output ranks a  target relative to other potential targets. Undervaluing secondary  economic impacts could result in a lower relative risk ranking that  underestimates the security risk to a target, or inversely, overvaluing  secondary economic impacts could result in overestimating the security  risk to a target. Recognizing the challenges with assessing secondary  economic impacts, Coast Guard officials said they are working with the  DHS Office of Risk Management and Analysis to study ways to more  accurately assess secondary economic impacts. Additionally, during the  course of our review the Coast Guard implemented a tool called IMPLAN  that has the potential to inform judgments of secondary economic impacts  by showing what the impact could be for different terrorist scenarios.
Limited time to complete assessments. Officials from 19 sectors (54  percent) told us that the lack of time to complete their annually required  vulnerability and consequence assessments is a key challenge and many  expressed that they believed their sector’s data suffered in quality as a  result. Each year, sectors are required to update and validate their risk  assessments for targets in their areas of responsibility, which can involve  site visits to port facilities and discussions with facility security officers to  obtain information on vulnerability and consequences. Officials from a  Gulf Coast sector noted that obtaining this information from facilities can  be challenging because of the number of facilities in the sector and the  time involved in meeting with each facility. Officials from an inland river  sector also noted that gathering data from certain facilities—such as  information on a chemical plant’s security enhancements or the expected  loss of life from a terrorist attack—is challenging because facilities may  not want to share proprietary information that could be damaging in the  hands of a competitor. As a result, it often takes additional visits, phone  calls, e-mails, and time to obtain this information. Officials from a  northeastern sector said that having the people and time to update  MSRAM data is their key challenge and completing the update is a heavy  lift because the update is required at the same time as several other  requirements, such as reviewing investment justifications for the Port  Security Grant Program. Coast Guard sector officials and one district  official we spoke with reported raising concerns to headquarters about the  time it takes to complete MSRAM assessments. Headquarters staff also  said they were looking into additional ways to make the assessment  process easier for sectors, such as providing job aids and examining the  possibility of completing the data update at different times in the year.
Limitations in modeling methodology—adaptive terrorist behavior.  There are inherent limitations in the overall methodology the Coast Guard  uses to model risk. For instance, MSRAM threat information does not  account for adaptive terrorist behavior, which is defined by the National  Research Council as an adversary adapting to the perceived defenses  around targets and redirecting attacks to achieve its goals. Accounting  for adaptive terrorist behavior could be modeled by making threat a  function of vulnerability and consequence rather than the MSRAM  formula which treats threat, vulnerability, and consequence as  independent variables.a critique of MSRAM raised by terrorism risk assessment experts. For  example, officials from the DHS Office of Risk Management and Analysis  have stressed the need to account for adaptive terrorist behavior in risk  models. In addition, DHS’s 2011 Risk Management Fundamentals  guidance states that analysts should be careful when calculating risk by  multiplying threats, vulnerabilities, and consequences (as MSRAM does),   Not accounting for adaptive terrorist behavior is  especially for terrorism, because of interdependencies between the three  variables. Coast Guard officials agreed with the importance of  accounting for adaptive terrorist behavior and with the risks of treating  threat, vulnerability, and consequence as independent variables. The  officials explained that although they did not design MSRAM to account  for adaptive terrorist behavior, they are working to develop the Dynamic  Risk Management Model, which will potentially address this issue.
For more information on network effects, see Gerald G. Brown, W. Matthew Carlyle,  Javier Salmerón, and Kevin Wood, Operations Research Department, Naval Postgraduate  School, Analyzing the Vulnerability of Critical Infrastructure to Attack and Planning  Defenses (Monterey, Calif.: 2005). initiatives to identify and document networked systems of targets that if  successfully attacked would have large ripple effects throughout the port or  local economy. Coast Guard officials agreed that assessing network effects  is a challenge and they are examining ways to meet this challenge.  However, the Coast Guard’s work in this area is still in its infancy and there  is uncertainty regarding the way in which the agency will move forward in  measuring network effects.
section_title: Additional Documentation Could Align MSRAM with NIPP Criteria for Documented and Defensible Risk Assessments
paragraphs: MSRAM is generally documented and defensible, but the Coast Guard  could improve its documentation of the model’s assumptions and other  sources of uncertainty, such as subjective judgments made by Coast  Guard analysts about threats, vulnerabilities, and consequences, and  how these assumptions and other sources of uncertainty affect MSRAM’s  results. The NIPP states that for a risk assessment methodology to be  documented, any assumptions and subjective judgments need to be  transparent to the individuals who are expected to use the results. For a  risk assessment methodology to be defensible, uncertainty associated  with consequence estimates and the level of confidence in the  vulnerability and threat estimates should also be communicated to users  of the results.
There are multiple assumptions and other sources of uncertainty in  MSRAM. For example, assumptions used in MSRAM include the  particular dollar value for a statistical life or the assumed dollar amount of  environmental damage resulting from oil or hazardous material spilled as  the result of a terrorist attack. MSRAM also relies on multiple subjective  judgments made by Coast Guard analysts, which mean a range of  possible values for risk calculated from the model. For example, to  assess risk in MSRAM, Coast Guard analysts make judgments regarding  such factors as the likelihood of success in interdicting an attack and the  number of casualties expected to result from an attack. These subjective  judgments are sources of uncertainty with implications that, according to  the NIPP and risk management best practices, should be documented  and communicated to decision makers.
MSRAM’s primary sources of documentation provide information on how  data are used to generate a risk estimate and information on some  assumptions, and the Coast Guard has made efforts to document and  reduce the number of assumptions made by the field-level user in order to  increase the consistency of MSRAM’s data. For example, the MSRAM  training and software manual states that MSRAM users are expected to  specify the assumptions they make in evaluating various attack modes  and provides assumptions for users to consider when scoring attack  scenarios, such as specifying the type and amount of biological agent  used in a biological attack scenario and assuming that attackers are  armed and suicidal in a boat bomb attack scenario.
While these documentation efforts are positive steps to reduce MSRAM  data subjectivity and increase data consistency, we found that the Coast  Guard has not documented all the sources of uncertainty associated with  threat, vulnerability, and consequence assessments and what  implications this uncertainty has for interpreting the results, such as an  identification of the highest-risk targets in a port. As a result, decision  makers do not know how robust the risk rankings of targets are and the  degree to which a list of high-risk targets could change given the  uncertainty in the risk model’s inputs and parameters. Moreover,  overlapping ranges of possible risk values caused by uncertainty could  have implications for strategic decisions or resource allocation, such as  allocating grant funding or targeting patrols. Overlapping ranges of risk  values due to uncertainty also underscores the importance of professional  judgment in decision making because risk models do not produce precise  outcomes that should be followed without a degree of judgment and  expertise.
According to the NIPP, the best way to communicate uncertainty will  depend on the factors that make the outcome uncertain, as well as the  amount and type of information that is available. The NIPP states that in  any given terrorist attack scenario there is often a range of outcomes that  could occur, such as a range of dollar amounts for environmental damage  or a range of values for a statistical life. For some incidents, the range of  outcomes is small and a single estimate may provide sufficient data to  inform decisions. However, if the range of outcomes is large, the scenario  may require additional specificity about conditions to obtain appropriate  estimates of the outcomes. Often, this means providing a range of  possible outcomes rather than a single point estimate. Coast Guard  officials agreed with the importance of documenting and communicating  the sources and implications of uncertainty for MSRAM’s risk estimates,  and noted that they planned to develop this documentation as part of an  internal MSRAM verification, validation, and accreditation (VV&A) process  that they expect to complete in the fall of 2011. According to the Coast  Guard, accreditation is an official determination that a model or simulation  is acceptable to use for a specific purpose. While this accreditation  process is expected to document the scope and limitations of MSRAM’s  capabilities and determine whether these capabilities are appropriate for  MSRAM’s current use, the Coast Guard’s draft accreditation plan does  not discuss how the Coast Guard plans to assess and document  uncertainty in its model or communicate those results to decision makers.
National Research Council of the National Academies, Review of the Department of  Homeland Security’s Approach to Risk Analysis. addressed and should address the structure of the model, the types and  certainty of the data, and how the model is intended to be used. Peer  reviews can also identify areas for improvement and can facilitate sharing  best practices. As we have previously reported, external peer reviews  cannot ensure the success of a model, but they can increase the  probability of success by improving the technical quality of projects and  the credibility of the decision-making process. MSRAM has been  reviewed twice—in 2010 by risk experts affiliated with the Naval  Postgraduate School and, to a lesser extent, in 2009 by CREATE at the  University of Southern California. The authors of the Naval Postgraduate  School report stated that their review was intended to validate and verify  the equations used in MSRAM, evaluate MSRAM’s quality control  procedures, and review the use of MSRAM outputs to manage risk. The  authors of the CREATE report stated that their review focused on  suggestions for improvement rather than a comprehensive evaluation,  and they suggested that the Coast Guard continue to seek feedback and  reviews from the risk and decision analysis community, as well as from  practitioners of other disciplines. Coast Guard officials told us that they  have generally benefited from reviews of MSRAM and have worked to  implement many of the resulting recommendations. Officials noted they  intend to pursue external reviews of MSRAM as part of the ongoing VV&A  process, but they have not identified who would be conducting the  reviews, or when the reviews would occur. As the Coast Guard’s risk  assessment model continues to evolve, the Coast Guard could benefit  from periodic external peer review to ensure that the structure and  outputs of the model are appropriate for its given uses and to identify  possible areas for improvement.
section_title: Coast Guard Has Used a Risk-Informed Approach to Manage Maritime Security Risk, but Challenges Hinder Sector Efforts
subsections: section_title: MSRAM Informs Several National-Level Risk Management Efforts
paragraphs: MSRAM is a security risk analysis and risk management tool and the  Coast Guard intends for it to be used to inform risk management  decisions at all levels of command. As such, in a May 2011 guidance  document, the Coast Guard set expectations for how MSRAM should be  used at the national and sector levels. At the national level, the Coast  Guard expects its offices to use MSRAM to support strategic plans,  policy, and guidance; to integrate MSRAM into maritime security  programs; and to ensure that sectors have adequate personnel ready to  perform MSRAM duties, among other goals.
Operational activities include conducting boat escorts, implementing positive control  measures—that is, stationing armed Coast Guard personnel in key locations aboard a  vessel to ensure that the operator maintains control—and providing a security presence  through various actions. (MSRO) program. By identifying the nation’s highest-risk maritime  targets, MSRAM helps establish the national MCIKR list, which sectors  use to complete their annually required number of MCIKR visits.  According to Coast Guard officials, MSRAM has aided in reducing the  MCIKR list from 740 assets to 324 assets and allowed the Coast Guard to  further prioritize within that more focused list of 324, since MSRAM  analysis demonstrated that a small number of assets make up the  majority of the nation’s risk.
MSRAM has also been used as a tool to inform resource allocation and  performance measurement, which is consistent with the Coast Guard’s  goals for MSRAM. For instance, risk-informed methods and processes or  models, such as MSRAM, are used in the Coast Guard’s annual Standard  Operational Planning Process, which establishes a standardized process to  apportion major assets, such as boats, aircraft, and deployable specialized  forces. Coast Guard officials said that MSRAM data supports the PWCS  mission in this process by demonstrating how risk is distributed  geographically. In addition, Coast Guard used MSRAM to support a  funding request for boats, personnel, and associated support costs to  assist with Coast Guard efforts to reduce the risk of certain dangerous  cargoes by escorting ships passing through coastal ports carrying cargoes  such as liquefied natural gas. MSRAM also supports resource allocation  through the Port Security Grant Program by informing the risk formula used  by DHS to allocate grant funding.
MSRAM data are also used in the  Coast Guard’s model for measuring its performance in the PWCS mission,  which is discussed in depth later in this report.
MSRAM has also supported strategic documents and efforts throughout  DHS. Specifically, the Coast Guard reported that MSRAM data are an  essential building block for a number of key strategic documents, such as  the National Maritime Strategic Risk Assessment, the National Maritime  Terrorism Threat Assessment, and the Combating Marine Terrorism  Strategic and Performance Plan, among others. In addition, the Coast  Guard uses MSRAM, among other inputs, to provide DHS with maritime  risk information for the Transportation Sector Security Risk Assessment  tool. DHS also reported that the Coast Guard has shared MSRAM- based identification of critical assets beyond the transportation system  with 13 of the 18 DHS critical infrastructure and key resource sectors.
For example, MSRAM has been used to assess the risk of some  chemical facilities and power plants.
section_title: MSRAM Has Informed Some Local-Level Risk Management Efforts, but Its Use Has Been Limited by Several Factors
paragraphs: MSRAM has been used to inform a variety of efforts at the sector level,  such as strategic planning, communication with port stakeholders, and  operational and tactical decision making, but its use for operational and  tactical risk management efforts has been limited by a lack of staff time,  the complexity of the MSRAM tool, and competing mission demands,  among other factors. The Coast Guard expects its 35 sectors, with  support from its nine districts, to integrate MSRAM data into strategic,  operational, and tactical plans, operations, and programs as necessary  and required, among other actions.
Based on results from our interviews with officials from all 35 Coast  Guard sectors, officials from 26 sectors (74 percent) reported finding  MSRAM moderately to very useful for informing strategic planning, which  includes developing portions of local Area Maritime Security Plans and  planning security exercises. sector reported using MSRAM to find the highest-risk areas in which to  conduct exercises. Further, lessons learned from the exercises are  incorporated into strategic plans, which officials said leads to planning  process improvements and overall better plans. However, officials from a  southeastern sector pointed out that MSRAM is a snapshot view of port  risk and therefore long-term strategic plans require additional information  from many sources.
Area Maritime Security Plans have been established pursuant to the Maritime  Transportation Security Act of 2002. Content requirements for the plans were established  by 33 C.F.R. § 103.505 and expanded by the Security and Accountability For Every Port  (SAFE Port) Act of 2006 to include a Salvage Response Plan. The plans are intended to  sponsor and support engagement with port community stakeholders to develop, test, and  when necessary, implement joint efforts for responding to and mitigating the effects of a  maritime transportation security incident. percent) said that MSRAM was moderately to very useful. For instance,  officials from a southeastern sector said that MSRAM is used to  communicate and justify additional security procedures. Further, during  annual compliance inspections, MSRAM data are discussed with facility  security officers and compared to security data that the facility security  officers have calculated. In addition, officials from a Gulf Coast sector  reported that MSRAM provides a convenient, objective way to  communicate risk to port security stakeholders, and stakeholders  appreciate that risk information from MSRAM is computer driven and  based on a rigorous process.
For informing sector operational and tactical decision making, such as  planning MSRO activities, developing local critical infrastructure lists, and  planning for special events, officials from 18 sectors (51 percent) reported  that MSRAM moderately or greatly provided them with the information  needed to make risk-informed decisions regarding port security.  Regarding planning MSRO activities, one eastern sector reported that  MSRAM was very helpful for identifying priority targets for MSRO patrols  and escorts. Regarding developing local critical infrastructure lists,  officials from an eastern sector said that since the sector has no assets  on the national MCIKR list, they were able to use MSRAM to generate a  local list to help determine patrols and other security efforts. Regarding  special event planning, officials from 16 sectors (45 percent) told us they  used MSRAM to determine where to allocate resources for special  events, such as the Fourth of July, dignitary visits, or political  conventions. For example, officials from an inland river sector said that  they used MSRAM to identify possible attack scenarios and to help  identify what security resources they should request to provide security  for a special event. See figure 1 for photographs of various Coast Guard  security-related activities that can be informed by MSRAM. In addition to  using MSRAM to inform maritime security decisions, officials from almost  every sector noted that they also assess and manage risk using other  tools or methods, such as the High Interest Vessel matrix, outreach to  port partners, working relationships with Area Maritime Security  Committees, or professional judgment.
Although officials from most sectors found that MSRAM provided useful  risk information for sector-level decision making, officials from 32 sectors  (91 percent) reported that their overall use of MSRAM data in managing  risk was hindered by a lack of staff time for data analysis, the complexity  of the MSRAM tool, or competing mission demands, among other things.  These challenges are discussed below.
Limited staff time for analyzing and using MSRAM. Officials from 21  sectors (60 percent) told us that limited staff time posed a challenge to  incorporating MSRAM into strategic, operational, and tactical planning  efforts. For example, officials from a northeastern sector said that a lack  of available staff time was one of the most significant limitations to  utilizing MSRAM. These officials stated that they would like to have  dedicated MSRAM personnel to develop the tool and make it useful on a  daily basis. They added that even though MSRAM had many capabilities,  they were unable to use it to its full capability because their port security  specialist—the primary user of MSRAM—was busy with other programs,  such as the Port Security Grant Program. Each of the port security  specialists from the three districts we interviewed—which encompass 15  sectors over the West Coast, East Coast, Gulf Coast, and Mississippi  River area—echoed the challenges with the level of sector resources for  MSRAM. For example, one district official stated that although Coast  Guard headquarters has dedicated MSRAM staff, there are no full-time  MSRAM analysts at the sector level. He added that each sector would  need a dedicated person for MSRAM and risk analysis to bring MSRAM  analysis into operational and tactical decision making.
Complexity of the MSRAM tool. Officials from 14 sectors (40 percent)  reported that MSRAM use has been limited because data outputs require  a substantial degree of analysis to use in decision making, or because the  MSRAM tool itself is not easy to use. Some of the challenges raised by  sectors that contribute to the complexity of the tool and interpreting its  outputs included keeping abreast of yearly changes to the MSRAM tool  and bridging knowledge gaps that occur when staff familiar with MSRAM  rotate or leave the sector. In its MSRAM core document, the Coast Guard  recognized that the frequent rotation of active duty personnel presents a  risk to both the consistency of the MSRAM risk scoring efforts and the  application of risk results.
Competing mission demands and resource constraints. Officials from  14 sectors (40 percent) reported that competing mission demands or  resource constraints limited the use of MSRAM. Specifically, officials from  11 sectors reported that MSRAM’s usefulness was limited by the fact that  it only considers risk in the PWCS mission, which is 1 of the Coast  Guard’s 11 statutorily required missions. For example, a Great Lakes  sector told us that while MSRAM identifies the risks in the sector, the  sector is limited in its ability to move assets to address those security  risks because the assets are also fulfilling other Coast Guard mission  requirements, such as search and rescue. Additionally, officials from 6  sectors said that limited resources, such as boats or personnel,  constrained their sectors’ ability to address the risks identified by  MSRAM. For example, officials from 2 inland river sectors said that  MSRAM identifies their security risks and demonstrates where they  should patrol and plan for special events, but that they do not have the  resources to carry out the plans. Further, officials from 1 of the inland  river sectors added that their response boats are often busy escorting the  Army Corps of Engineers or engaged in flood relief efforts. This leaves  the work of security patrols to the local harbor patrol, which the officials  said does not have the same capabilities, in terms of boats and weapons,  as the Coast Guard.
Other challenges. Sector officials also identified other challenges with  using MSRAM for informing decision making. Specifically, officials from  16 sectors (45 percent) said that MSRAM would be more useful if it was  linked to other Coast Guard data systems, such as the Coast Guard’s  inspections database, or if MSRAM was integrated into the sector  command center. For example, officials from an east coast sector told us  that they would like to see MSRAM linked to other databases in the sector  command center, such as the Coast Guard’s vessel tracking system.  Similarly, officials from a west coast sector said that integrating MSRAM  into the Coast Guard’s inspections database would keep MSRAM  continually updated and reflective of inspection results. Further, the  command center has to consider other mission response needs, such as  for pollution incidents or search and rescue, among others, and if  MSRAM was integrated into the sector command center it could be used  more in day-to-day operations. In addition, officials from 5 sectors noted  that MSRAM does not capture dynamic risk, which limits its ability to  inform daily decisions at the sector level. For instance, officials from a  Gulf Coast sector said that they did not use MSRAM on a daily basis to  allocate resources because daily fluctuations in vessel and barge risk are  their greatest concern and this risk is not currently captured in MSRAM.  The sectors that raised these issues believed that linking MSRAM into  other data systems, integrating MSRAM into the command center, and  having MSRAM account for dynamic risks could contribute to making its  data more accurate, robust, and useful for decision making.
section_title: Coast Guard Has Taken Steps to Address Challenges, but Could Increase MSRAM’s Use by Expanding Training Opportunities
paragraphs: Coast Guard headquarters officials told us that they were aware of the  challenges field-level MSRAM users were facing and have taken some  steps to address them, but providing additional training could help  integrate MSRAM throughout sector decision making. The Coast Guard’s  current actions to address MSRAM user challenges include assessing the  feasibility of adding additional risk analyst staff, increasing the data’s  usability, developing decision-supporting modules, and providing training.  These actions are described below.
Examining the feasibility of dedicated risk analysts. Presently, there  is no dedicated risk analyst or MSRAM analyst position at the sector  level, but headquarters officials told us in June 2011 that they are  examining the feasibility of assigning additional port security specialists to  the field and submitted a resource proposal for the additional staff.  According to a senior Coast Guard budget official, given competing  priorities and a constrained resource environment, it is unclear when or if  this resource proposal will be funded.
Deploying MSRAM to sector command centers. To help make  MSRAM more dynamic and increase its usability, the Coast Guard is  piloting an Enterprise Geographic Information System (EGIS) display for  sector command centers, which layers facility and vessel locations onto a  satellite-based map and visually displays changing risk as vessels move  into and out of ports. Officials from 7 sectors that participated in or were  familiar with the initial EGIS test group reported that the functionality was  very useful and had the potential to substantially increase MSRAM’s use  for sector risk management efforts. In addition, headquarters officials told  us in June 2011 that efforts were under way to integrate MSRAM into the  Coast Guard’s inspections database, which would allow MSRAM to be  continually updated and reflective of year-round facility and vessel  inspection results.
Developing risk management modules. To assist with incorporating  risk assessment information into decision making, in the fall of 2008, the  Coast Guard began developing risk management modules within MSRAM  that are able to provide specific types of analyses, such as comparing  alternative security strategies. We asked officials from all 35 sectors their  views on four modules—the Alternatives Evaluation Module, the  Simplified Reporting Interface, the Daily Risk Profile, and the Risk  Management Module. Sectors had mixed views on the utility of these  modules. Specifically, officials from 14 sectors (40 percent) found the  Alternatives Evaluation module very useful and cited such uses as  evaluating Port Security Grant Program proposals and planning security  for special events, and officials from 15 sectors (42 percent) found the  Simplified Reporting Interface very useful for communicating risk  information to port partners. However, with respect to the other two  modules—the Daily Risk Profile and Risk Management Module—officials  from 2 sectors (5 percent) found the Daily Risk Module very useful and  officials from 3 sectors (8 percent) found the Risk Management Module  very useful. For both modules, officials from 18 sectors (51 percent)  reported that either they had not seen them or they were aware of the  modules but did not have the time or training, among other reasons, to  use them. Many of the modules are new and headquarters and some  sector officials reported that they expected the modules would be more  useful in the future as sectors gained familiarity with them through  additional exposure and the annual MSRAM training.
Providing training. While the Coast Guard offers annual MSRAM  training, officials from 25 sectors (71 percent) identified areas of the  training for improvement, which the Coast Guard could do more to  address. Specifically, officials from these sectors said that increasing  the number of people who take MSRAM training, providing MSRAM  training to command-level staff or senior management, and offering  training on how to conduct risk analysis to inform decision making, among  other things, would help integrate MSRAM throughout sector decision- making processes. Since MSRAM is a collateral duty, MSRAM training is  not part of any Coast Guard personnel’s required training curriculum.  However, Coast Guard guidance from May 2011 states that area, district,  and sector commanders are responsible for ensuring that adequate  numbers of appropriate personnel are trained in MSRAM. Only one sector  did not, at the time of our interview, have at least one staff person trained  in MSRAM. Officials from a Gulf Coast sector said that the training  provided on the MSRAM tool itself is good, but the training does not teach  the skills needed to make decisions in the field. Officials from a Great  Lakes sector suggested that the Coast Guard develop an advanced  course on how to use MSRAM to inform operational decisions. Officials  from a southeastern sector added that the Coast Guard provides  guidance on how to assess risks using MSRAM, but needs to provide  more training on how to communicate MSRAM results and how those  results can be used. In addition, a sector commanding officer who  participated in one of our interviews told us that he was provided minimal  training on MSRAM and wanted to understand more about how it can be  used to support command-level decisions.
MSRAM has the capability of informing operational, tactical, and resource  allocation decisions at all levels of a sector, but the Coast Guard has  generally provided MSRAM training to a limited number of sector staff  with specific MSRAM risk assessment responsibilities, such as port  security specialists, rather than sector staff who may have command or  management responsibilities where MSRAM may apply. Coast Guard  headquarters officials said that this was because of limited resources to  provide training for numerous sector personnel and variations in how  MSRAM responsibilities are managed at different sectors. Standards for  Internal Control in the Federal Government states that effective  management of an organization’s workforce is essential to achieving  results. Further, only when the right personnel for the job are on board  and are provided the right training and tools, among other things, is  operational success possible. To this end, management should ensure  that training is aimed at developing and retaining employee skill levels to  meet changing organizational needs. Coast Guard headquarters  officials agree that providing MSRAM training to additional sector staff,  particularly those with command and management responsibilities, would  be valuable. Such training on how MSRAM can be used at all levels of  command for risk-informed decision making—including how MSRAM can  assist with the selection of different types of security measures to address  areas of risk and the evaluation of their impacts—could further the Coast  Guard’s efforts to implement its risk management framework and meet its  goal to institutionalize MSRAM as the risk management tool for maritime  security.
section_title: Coast Guard Measures Risk Reduction but Has Faced Challenges Using This Measure to Inform Decisions
subsections: section_title: Coast Guard Developed a Measure to Report Performance in Reducing Risk
paragraphs: The Coast Guard developed a performance measure and supporting  model to measure and report its overall performance in reducing maritime  security risk. This measure identifies the percentage reduction of maritime  security risk, subject to Coast Guard influence, resulting from various  Coast Guard actions. The Coast Guard considers this performance  measure its key outcome measure for its PWCS mission. According to  DHS’s Risk Management Fundamentals and the NIPP, it is crucial that a  process of performance measurement be established to evaluate whether  actions taken ultimately achieve the intended performance objective, such  as reducing risk. This is important not only in evaluating program  performance but also in holding the organization accountable for  progress. We have also previously reported on the importance of  developing outcome-based performance goals and measures as part of  results management efforts. From fiscal years 2006 to 2010, the Coast  Guard annually reported reducing from 15 to 31 percent of the maritime  risk it is responsible for, in each year either meeting or exceeding its  target. For fiscal years 2011 and 2012, the Coast Guard’s planned  performance targets are to reduce more than 44 percent of the maritime  security risk for which it is responsible.
To measure how its actions have reduced risk, the Coast Guard  developed a model that uses a two-step approach. The first step is to  estimate the total amount of terrorism risk that exists in the maritime  domain, in the absence of any Coast Guard activities. This is referred to  as raw risk, and this information comes primarily from MSRAM. second step relies on an elicitation process whereby Coast Guard subject  matter experts estimate how various security activities and operations,  maritime domain awareness programs, and regulatory structures— referred to by the Coast Guard as regimes—that the Coast Guard has  implemented have reduced risk to U.S. ports and waterways. This step  involves Coast Guard subject matter experts assessing the probability of  these Coast Guard efforts failing to prevent a successful terrorist attack  for 16 potential maritime terrorist attack scenarios.
Information also comes from DHS’s Risk Analysis Process for Informed Decision Making  (RAPID) project, which is designed to provide strategic planning guidance and support  resource allocation decisions at the DHS level.
According to DHS’s Risk Management Fundamentals, elicitations involve using  structured questions to gather information from individuals with in-depth knowledge of  specific areas or fields. missions, such as search and rescue, there is not a rich historical data set  of maritime terrorism incidents that the Coast Guard can use to measure  its actual performance. In other words, in the absence of an actual  domestic maritime terrorism event, the Coast Guard uses internal subject  matter experts to estimate risk reduction as a proxy measure of  performance—an attempt to measure performance against a terrorism  incident that did not occur.
section_title: Coast Guard Faces Challenges Using the Risk Reduction Performance Measure to Inform Decision Making
paragraphs: The Coast Guard’s efforts to develop an outcome measure to quantify the  impact its actions have had on risk is a positive step. However, the use of  the measure has been limited, and even with recent improvements, the  Coast Guard faces challenges using this measure to inform decision  making. Performance goals and measures are intended to provide  Congress and agency management with information to systematically  assess a program’s strengths, weaknesses, and performance. Thus,  measures should provide information for management decision making.  Coast Guard officials explained that the primary purpose of the risk  reduction measure has been for external performance reporting, and to a  more limited extent for informing strategic decision making and for  conducting internal analysis of performance to identify areas for  improvement. Specifically, officials said the measure has been used to  compare risk across maritime terrorism scenarios and compare those  results to other studies and analysis on maritime terrorism scenarios,  which provided information on whether PWCS activities were  appropriately balanced to address those risks. However, Coast Guard  officials stated that over time, internal and external reviews identified  limitations in the risk reduction measure, such as not allowing for  comparisons of performance across sectors.
Recognizing these limitations, in 2010, the Coast Guard made  improvements to the risk reduction model intended to enhance its utility  for management decision making and to provide a more accurate  measure of risk reduction. For example, the updated model includes  information on the locations of Coast Guard assets and potential targets,  which can be used to calculate the probability that Coast Guard assets  will be able to intercept attacks. The Coast Guard also improved the  elicitation techniques by which subject matter experts provided their  estimates of Coast Guard risk reduction performance, and expanded the  size and diversity of the subject matter experts involved in the elicitation   According to Coast Guard officials, these improvements have  process. made the measure and supporting model more useful for informing  strategic decisions by allowing, for example, the ability to calculate risk  reduction at the sector, district, area, and national levels and the risk  reduction value of each element of the Coast Guard’s strategy. In other  words, the updated model is able to show the risk reduction value of  Coast Guard operational assets, such as small boats or helicopters,  compared with regime activities, such as regulation enforcement. This  information can help inform resource allocation decisions because it could  identify which actions provide the greatest risk-reduction, according to  these officials. The Coast Guard plans to use the updated model to  measure its performance in reducing risk for the 2011 fiscal year.
According to the Coast Guard, in 2009 a total of 26 subject matter experts were used,  mostly from headquarters. In 2010, a total of 46 subject matter experts were used coming  from headquarters, areas, districts, sectors, and operational units. making. For example, given the inherent uncertainties in estimating risk  reduction, it is unclear if a measure of risk reduction would provide  meaningful performance information for tracking progress against goals  and performance over time. According to our performance measurement  criteria, to be able to assess progress toward the achievement of  performance goals, the measures used must be reliable and valid.  Reliability refers to the precision with which performance is measured,  while validity is the extent to which the measure adequately represents  actual performance. Therefore, the usefulness of agency performance  information depends to a large degree on the reliability of performance  data. We have also reported that decision makers must have assurance  that the program data being used to measure performance are sufficiently  reliable and valid if the data are to inform decision making. Although the  Coast Guard has taken steps to improve the quality of the supporting  model to provide a more accurate measure, estimating risk reduction is  inherently uncertain and this measure is based on largely subjective  judgments of Coast Guard personnel, and therefore the risk reduction  results reported by the Coast Guard are not based on measurable or  observable activities. As a result, it is difficult to independently verify or  assess the validity or appropriateness of the judgments or to determine if  this is an accurate measure of Coast Guard performance in the PWCS  mission. However, Coast Guard officials told us that they believe these  reported results provide a useful proxy measure of Coast Guard  performance, and noted that this is one of several metrics the Coast  Guard uses to assess performance in the PWCS mission.
According to DHS’s Risk Management Fundamentals, it is also important  to be transparent about assumptions and key sources of uncertainty, so  that decision makers are informed of the limitations of the risk information  provided by the model. In its 2009 review of the risk reduction model,  CREATE at the University of Southern California stated that it seemed  likely that the model ignored important uncertainties and implied  incorrectly high precision of risk estimates. Furthermore, OMB’s  Updated Principles for Risk Analysis notes that because of the inherent  uncertainties associated with estimates of risk, presentation of a single  risk estimate may be misleading and provide a false sense of precision.  OMB suggests that when a quantitative characterization of risk is  provided, a range of plausible risk estimates should also be provided.  From fiscal years 2006 to 2010, the Coast Guard reported the risk  reduction measure as a specific risk reduction number rather than as a  range of plausible risk reduction estimates. The Coast Guard official  responsible for this measure told us this was because the previous risk  reduction model was not capable of producing a range of plausible risk  reduction estimates. The official noted that while the new risk reduction  model—which will be used to report results for fiscal year 2011—is  capable of producing a range of estimated risk reduction, the Coast  Guard will continue to report the risk reduction measure as a single  number because the DHS data system for performance reporting does  not accept ranges—only numerical values. However, the official added  that there is value in reporting a range of risk reduction and officials are  considering a transition to a range of estimated reduction for the PWCS  mission in future years. One alternative could be to report the percentage  of risk reduced as a single number, but having an explanatory note  indicating the range of plausible risk reduction estimates. Using a risk  reduction measure that more accurately reflects performance  effectiveness can give Coast Guard leaders and Congress a better sense  of progress toward goals, which can support efforts to identify areas for  improvement.
DHS officials have also raised some questions about the risk reduction  measure. Recently, DHS determined that the Coast Guard’s risk  reduction measure was not appropriate for inclusion as a DHS strategic  performance measure and has designated it as a management measure.  According to DHS, a strategic measure is designed to communicate  achievement of strategic goals and objectives and be readily  understandable to the public, and a management measure is designed to  gauge program results and tie to resource requests and be used to  support achievement of strategic goals. According to a senior DHS  official, in 2010, DHS leadership reviewed all existing department  measures and made decisions about which measures they believed were  clearly tied to the DHS Quadrennial Homeland Security Review missions  and were easily understandable by the public. This official noted that  based on this review, DHS leadership did not feel the risk reduction  measure and its methodology would be easily understandable by the  public and therefore did not designate the measure as a strategic  measure. As a result, the risk reduction measure will not be included in  DHS’s annual performance plan, formally published with the Annual  Performance Report, because this report only includes the smaller set of  strategic measures. However, this official noted that the risk reduction  measure is important as one piece of information to manage risk and is  considered to be part of the full suite of DHS performance measures, and  will continue to be published in the Coast Guard’s strategic context that is  submitted with DHS’s Annual Performance Report.
section_title: Conclusions
paragraphs: The Coast Guard has invested substantial effort incorporating risk  management principles into its security priorities and investments, and  continues to proactively strengthen its assessment, management, and  evaluation practices. As a result, the Coast Guard’s risk assessments and  risk model are generally sound and in alignment with DHS standards.  However, there are some additional actions that the Coast Guard could  take to further its risk management approach by facilitating a wider use of  risk information and making the results more valuable to the users. For  example, since risk management is a tool for informing policymakers’  decisions about assessing risks, allocating resources, and taking actions  under conditions of uncertainty, the Coast Guard could better document  and communicate the uncertainty or confidence levels of its risk  assessment results, including any implications that the uncertainty may  have for decision makers. This added information would allow Coast  Guard decision makers to prioritize strategies, tactics, and long-term  investments with greater insight about the range of likely results and  associated trade-offs with each decision. Additional information would  also allow external reviewers of the risk model to reach the most  appropriate conclusions or provide the most useful improvement  recommendations through periodic reviews. The Coast Guard could also  enhance the risk-informed prioritization of its field-level strategies,  operations, and tactics by ensuring that risk management training is  expanded to multiple levels of Coast Guard decision makers at the sector  level, including command-level personnel. Expanding training on how  MSRAM could be used at all levels of command for risk-informed decision  making—including how MSRAM can assist with the selection of different  types of security measures and the evaluation of their impacts—would  further the Coast Guard’s efforts to implement its risk management  framework and meet its goal of institutionalizing MSRAM as the risk  management tool for maritime security. Finally, accurately representing  performance results is important and the Coast Guard could more  accurately convey its risk reduction performance measure by reporting  risk reduction results as a range rather than a point estimate. Presenting  risk reduction as a single number without a corresponding range of  uncertainty could hamper Coast Guard efforts to identify areas for  improvement. Taking these steps would make the Coast Guard’s risk  management approach even stronger.
section_title: Recommendations for Executive Action
paragraphs: To help the Coast Guard strengthen MSRAM and better align it with NIPP  risk management guidance, as well as facilitate the increased use of  MSRAM across the agency, we recommend that the Commandant of the  Coast Guard take the following three actions:  (1)  Provide more thorough documentation related to key assumptions  and sources of uncertainty within MSRAM and inform users of any  implications for interpreting the results from the model.  (2)  Make MSRAM available to appropriate parties for additional external  peer review.  (3)  Provide additional training for sector command staff and others  involved in sector management and operations on how MSRAM can  be used as a risk management tool to inform sector-level decision  making.
To improve the accuracy of the risk reduction measure for internal and  external decision-making, we recommend that the Commandant of the  Coast Guard take action to report the results of the risk reduction  measure as a range rather than a point estimate.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS and the Coast Guard on  October 17, 2011, for review and comment. DHS provided written  comments, which are reprinted in appendix II. DHS and the Coast Guard  concurred with the findings and recommendations in the report, and  stated that the Coast Guard is taking actions to implement our  recommendations.
The Coast Guard concurred with our first recommendation that it provide  more thorough documentation related to key assumptions and sources of  uncertainty within MSRAM. Specifically, the Coast Guard stated that the  documentation of uncertainty is part of the ongoing MSRAM VV&A  process, and that the Coast Guard will continue to work with the DHS  Office of Risk Management and Analysis in developing a feasible and  deployable model that will benefit field-level security operations. These  actions should improve the Coast Guard’s ability to document and inform  MSRAM users of any implications for interpreting results from the model,  thereby addressing the intent of our recommendation.
Regarding the second recommendation that the Coast Guard make  MSRAM available to appropriate parties for additional external peer  review, the Coast Guard concurred. The Coast Guard stated that external  peer review is part of the ongoing MSRAM VV&A process, and that  additional external peer review will be part of an independent verification  and validation of MSRAM expected to be completed in the fall of 2012.   Such actions should address the intent of the recommendation.
Regarding the third recommendation that the Coast Guard provide  additional training for sector command staff and others involved in sector  management on how MSRAM can be used as a risk management tool,  the Coast Guard concurred. Specifically, the Coast Guard stated that  MSRAM is part of the Coast Guard’s contingency planning course, and  the Coast Guard will explore other opportunities to provide risk training to  sector command staff, including online and webinar training opportunities.  Such actions, once implemented, should address the intent of the  recommendation.
Finally, the Coast Guard also concurred with the fourth recommendation  to take action to report the results of the risk reduction measure as a  range rather than a point estimate. The Coast Guard stated that it is  currently limited by the DHS data reporting system with regard to the  format of presenting performance targets and results, but noted that it is  currently working with DHS to determine options for reporting risk as a  range. Such action, when fully implemented, should address the intent of  the recommendation.
DHS and the Coast Guard also provided us with technical comments,  which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of  Homeland Security, appropriate congressional committees, and other  interested parties. In addition, the report will be available at no charge on  the GAO website at http://www.gao.gov.
If you or your staff have any further questions about this report, please  contact me at (202) 512-9610 or caldwells@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. Key contributors are listed in appendix III.
section_title: Appendix I: Risk Management Framework
paragraphs: To provide guidance to agency decision makers, we developed a risk  management framework which is intended to be a starting point for  applying risk-informed principles. Our risk management framework,  shown in figure 2, entails a continuous process of managing risk through  a series of actions, including setting strategic goals and objectives,  assessing risk, evaluating alternatives, selecting initiatives to undertake,  and implementing and monitoring those initiatives.
Setting strategic goals, objectives, and constraints is a key first step in  applying risk management principles and helps to ensure that  management decisions are focused on achieving a purpose. Risk  assessment, an important element of a risk-informed approach, helps  decision makers identify and evaluate potential risks so that  countermeasures can be designed and implemented to prevent or  mitigate the effects of the risks. Risk assessment is a qualitative  determination, quantitative determination, or both of the likelihood of an  adverse event occurring and the severity, or impact, of its consequences.  Risk assessment in a homeland security application involves assessing  three key components—threat, vulnerability, and consequence. A threat  assessment is the identification and evaluation of adverse events that can  harm or damage an asset. A vulnerability assessment identifies  weaknesses in physical structures, personal protection systems,  processes, or other areas that may be exploited. A consequence  assessment is the process of identifying or evaluating the potential or  actual effects of an event, incident, or occurrence. Information from these  three assessments contributes to an overall risk assessment that  characterizes risks, which can provide input for evaluating alternatives  and prioritizing security initiatives. The risk assessment element in the  overall risk management cycle informs each of the remaining steps of the  cycle. Alternatives evaluation addresses the evaluation of risk reduction  methods by consideration of countermeasures or countermeasure  systems and the costs and benefits associated with them. Management  selection addresses such issues as determining where resources and  investments will be made, the sources and types of resources needed,  and where those resources would be targeted. The next phase in the  framework involves the implementation of the selected countermeasures.  Following implementation, monitoring is essential to help ensure that the  entire risk management process remains current and relevant and reflects  changes in the effectiveness of the alternative actions and the risk  environment in which it operates. Program evaluation is an important tool  for assessing the efficiency and effectiveness of the program. As part of  monitoring, consultation with external subject area experts can provide a  current perspective and an independent review in the formulation and  evaluation of the program.
The National Infrastructure Protection Plan (NIPP), originally issued by  the Department of Homeland Security (DHS) in 2006 and updated in  2009, includes a risk analysis and management framework, which, for the  most part, mirrors our risk management framework. This framework  includes six steps—set goals and objectives; identify assets, systems,  and networks; assess risks; prioritize; implement programs; and measure  effectiveness. The NIPP is DHS’s base plan that guides how DHS and  other relevant stakeholders should use risk management principles to  prioritize protection activities. In 2009, DHS updated the NIPP to, among  other things, increase its emphasis on risk management, including an  expanded discussion of risk management methodologies and discussion  of a common risk assessment approach that provided core criteria for  these analyses. Beyond the NIPP, DHS has issued additional risk  management guidance and directives. For example, in January 2009  DHS published its Integrated Risk Management Framework, which,  among other things, calls for DHS to use risk assessments to inform  decision making. In April 2011, DHS issued its Risk Management  Fundamentals, which establishes specific doctrine and guidance for risk  management across DHS.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Dawn Hoff, Assistant Director  and Adam Hoffman, Analyst-in-Charge, managed this assignment. Chuck  Bausell, Charlotte Gamble, and Grant Sutton made significant  contributions to this report. Colleen McEnearney provided assistance with  interviews and data analysis. Michele Fejfar assisted with design,  methodology, and data analysis. Jessica Orr provided assistance with  report development, and Geoff Hamilton provided legal assistance.
section_title: Related GAO Products
paragraphs: Port Security Grant Program: Risk Model, Grant Management, and  Effectiveness Measures Could Be Strengthened. GAO-12-47.  Washington, D.C.: November 17, 2011.
Maritime Security: Progress Made but Further Actions Needed to Secure  the Maritime Energy Supply. GAO-11-883T. Washington, D.C.:   August 24, 2011.
Maritime Security: DHS Progress and Challenges in Key Areas of Port  Security. GAO-10-940T. Washington, D.C.: July 21, 2010.
Maritime Security: Varied Actions Taken to Enhance Cruise Ship  Security, but Some Concerns Remain. GAO-10-400. Washington, D.C.:  April 9, 2010.
Critical Infrastructure Protection: Update to National Infrastructure  Protection Plan Includes Increased Emphasis on Risk Management and  Resilience. GAO-10-296. Washington, D.C.: March 5, 2010.
Transportation Security: Comprehensive Risk Assessments and Stronger  Internal Controls Needed to Help Inform TSA Resource Allocation.   GAO-09-492. Washington, D.C.: March 27, 2009.
Department of Homeland Security: Progress Report on Implementation of  Mission and Management Functions. GAO-07-454. Washington, D.C.:  August 17, 2007.
Risk Management: Further Refinements Needed to Assess Risks and  Prioritize Protective Measures at Ports and Other Critical Infrastructure.  GAO-06-91. Washington, D.C.: December 15, 2005.
Managing for Results: Enhancing Agency Use of Performance  Information for Management Decision Making. GAO-05-927. Washington,  D.C.: September 9, 2005.
Homeland Security: Summary of Challenges Faced in Targeting  Oceangoing Cargo Containers for Inspection. GAO-04-557T. Washington  D.C.: March 31, 2004.
Managing for Results: Challenges Agencies Face in Producing Credible  Performance Information. GAO/GGD-00-52. Washington, D.C.:   February 4, 2000.
The Results Act: An Evaluator’s Guide to Assessing Agency Annual  Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1998.