Critical Infrastructure Protection: Preliminary Observations on DHS Efforts to Assess Chemical Security Risk and Gather Feedback on Facility Outreach
Released: 2013-03-14T00:00:00

Letter

Background

Section 550 of the DHS appropriations act for fiscal year 2007 requires DHS to issue regulations establishing risk-based performance standards for the security of facilities that the Secretary determines to present high levels of security risk, among other things. The CFATS rule was published in April 2007, and appendix A to the rule, published in November 2007, listed 322 chemicals of interest and the screening threshold quantities for each. ISCD has direct responsibility for implementing DHS’s CFATS rule, including assessing potential risks and identifying high-risk chemical facilities, promoting effective security planning, and ensuring that final high-risk facilities meet applicable standards through site security plans approved by DHS. From fiscal years 2007 through 2012, DHS dedicated about $442 million to the CFATS program. During fiscal year 2012, ISCD was authorized 242 full-time- equivalent positions.

ISCD uses a risk assessment approach to develop risk scores to assign chemical facilities to one of four final tiers. Facilities placed in one of these tiers (tier 1, 2, 3, or 4) are considered to be high risk, with tier 1 facilities considered to be the highest risk. According to an ISCD document that describes how ISCD develops its CFATS risk score, the risk score is intended to be derived from estimates of consequence (the adverse effects of a successful attack), threat (the likelihood of an attack), and vulnerability (the likelihood of a successful attack, given an attempt). ISCD’s risk assessment approach is composed of three models, each based on a particular security issue: (1) release, (2) theft or diversion, and (3) sabotage, depending on the type of risk associated with the 322 chemicals. Once ISCD estimates a risk score based on these models, it assigns the facility to a final tier.

ISCD Has Assigned Thousands of Facilities to Tiers, but ISCD’s Approach to Risk Assessment Does Not Reflect All Risk Elements

ISCD Has Tiered Thousands of High-Risk Chemical Facilities and Resolved Some Problems Using Its Risk Assessment Approach to Assign Tiers

In July 2007, ISCD began reviewing information submitted by the owners and operators of approximately 40,000 facilities. By January 2013, ISCD had designated about 4,400 of the 40,000 facilities as high risk and thereby covered by the CFATS rule. ISCD had assigned about 3,500 of those facilities to a final tier, of which about 90 percent were tiered because of the risk of theft or diversion. The remaining 10 percent were tiered because of the risk of release or the risk of sabotage.

Over the last 2 years, ISCD has identified problems with the way the release chemicals model assigns chemical facilities to tiers and has taken or begun to take action to address those problems. In February 2011, ISCD found that some chemical facilities had been placed in an incorrect final tier because this model included incorrect data about the release of high-risk chemicals of interest. In June 2011, ISCD officials adjusted the model, which resulted in lowering the tier for about 250 facilities, about 100 of which were subsequently removed from the CFATS program. In October 2012, ISCD officials stated that they had uncovered another defect that led the model to exclude population density calculations for about 150 facilities in states or U.S. territories outside the continental United States, including Alaska, Hawaii, Puerto Rico, and Guam. In February 2013, ISCD officials said that they had made adjustments to the model to resolve this issue and do not expect any facilities’ tier will change due to this issue.

ISCD’s Risk Assessment Approach Does Not Consider All Elements of Risk

Our preliminary analyses indicates that the tiering approach ISCD uses to assess risk and assign facilities to final tiers does not consider all of the elements of risk associated with a terrorist attack involving certain chemicals. According to the NIPP, which, among other things, establishes the framework for managing risk among the nation’s critical infrastructure, risk is a function of three components—consequence, threat, and vulnerability—and a risk assessment approach must assess each component for every defined risk scenario. Furthermore, the CFATS rule calls for ISCD to review consequence, threat, and vulnerability information in determining a facility’s final tier. However, ISCD’s risk assessment approach does not fully consider all of the core criteria or components of a risk assessment, as specified by the NIPP, nor does it comport with parts of the CFATS rule.  Consequence. The NIPP states that at a minimum, consequences should focus on the two most fundamental components—human consequences and the most relevant direct economic consequences. The CFATS rule states that chemical facilities covered by the rule are those that present a high risk of significant adverse consequences for human life or health, or critical economic assets, among other things, if subjected to terrorist attack, compromise, infiltration, or exploitation. Our review of ISCD’s risk assessment approach and discussions with ISCD officials shows that the approach is currently limited to focusing on one component of consequences—human casualties associated with a terrorist attack involving a chemical of interest—and does not consider consequences associated with economic criticality. ISCD officials said that the economic consequences part of their risk-tiering approach will require additional work before it is ready to be introduced. In September 2012, ISCD officials stated that they had engaged Sandia National Laboratories to examine how ISCD could gather needed information and determine the risk associated with economic impact, but this effort is in the initial stages, with an expected completion date of June 2014. ISCD officials added they are uncertain about how Sandia’s efforts will affect their risk assessment approach.  Threat. ISCD’s risk assessment approach is also not consistent with the NIPP because it does not consider threat for the majority of regulated facilities. According to the NIPP, risk assessments should estimate threat as the likelihood that the adversary would attempt a given attack method against the target. The CFATS rule requires that, as part of assessing site vulnerability, facilities conduct a threat assessment, which is to include a description of the internal, external, and internally assisted threats facing the facility and that ISCD review site vulnerability as part of the final determination of a facility’s tier. Our review of the models and discussions with ISCD officials shows that (1) ISCD is inconsistent in how it assesses threat using the different models because while it considers threat for the 10 percent of facilities tiered because of the risk of release or sabotage, it does not consider threat for the approximately 90 percent of facilities that are tiered because of the risk of theft or diversion; and (2) ISCD does not use current threat data for the 10 percent of facilities tiered because of the risk of release or sabotage. ISCD did not have documentation to show why threat had not been factored into the formula for approximately 90 percent of facilities tiered because of the risk of theft or diversion. However, ISCD officials pointed out that the cost of adding a threat analysis for these facilities might outweigh the benefits of doing so. ISCD officials said that given the complexity of assessing threat for theft or diversion, they are considering reexamining their approach. ISCD officials also said that they are exploring how they can use more current threat data for the 10 percent of facilities tiered because of the risk of release or sabotage.  Vulnerability. ISCD’s risk assessment approach is also not consistent with the NIPP because it does not consider vulnerability when developing risk scores. According to the NIPP, risk assessments should identify vulnerabilities, describe all protective measures, and estimate the likelihood of an adversary’s success for each attack scenario. Similar to the NIPP, the CFATS rule calls for ISCD to review facilities’ security vulnerability assessments as part of its risk-based tiering process. This assessment is to include the identification of potential security vulnerabilities and the identification of existing countermeasures and their level of effectiveness in both reducing identified vulnerabilities and meeting the aforementioned risk-based performance standards. Our review of the risk assessment approach and discussions with ISCD officials shows that the security vulnerability assessment contains numerous questions aimed at assessing vulnerability and security measures in place but the information is not used to assign facilities to risk-based tiers. ISCD officials said they do not use the information because it is “self- reported” by facilities and they have observed that it tends to overstate or understate vulnerability. As a result, ISCD’s risk assessment approach treats every facility as equally vulnerable to a terrorist attack regardless of location and on-site security. ISCD officials told us that they consider facility vulnerability, but at the latter stages of the CFATS regulatory process particularly with regard to the development and approval of the facility site security plan.

ISCD Has Begun to Take Actions to Examine How Its Approach Could Be Enhanced

Our preliminary work indicates that ISCD has begun to take some actions to examine how its risk assessment approach can be enhanced. For example, in addition to engaging Sandia National Laboratories to develop the framework for assessing economic consequences previously discussed, ISCD has commissioned a panel of subject matter experts to examine the strengths and weaknesses of its current risk assessment approach. ISCD officials stated that the panel’s work is intended to focus on whether ISCD is heading in the right direction, and they view it as a preliminary assessment. According to ISCD’s task execution plan, the panel is to provide actionable recommendations on potential improvements to the CFATS models, but the panel is not to develop alternative CFATS models or formally validate or verify the current CFATS risk assessment approach—steps that would analyze the structure of the models and determine whether they calculate values correctly. In February 2013, after the panel was convened, ISCD officials stated that they provided information to the panel about various issues that they might want to consider, among them, (1) how to address vulnerability in the models given ISCD concerns about data quality, and (2) what the appropriate variables to use, if any, are for threats associated with theft or diversion, as discussed earlier.

We believe that ISCD is moving in the right direction by commissioning the panel to identify the strengths and weaknesses of its risk assessment approach, and the results of the panel’s work could help ISCD identify issues for further review and recommendations for improvement. Given the critical nature of ISCD’s risk assessment approach in laying the foundation for further regulatory steps in improving facility security—such as the development and approval of facility site security plans—it is important that its approach for assigning facilities to tiers is complete within the NIPP risk management framework and the CFATS rule. Once ISCD’s develops a more complete approach for assessing risk it would then be better positioned to commission an independent peer review. In our past work, we reported that peer reviews are a best practice in risk management and that independent expert review panels can provide objective reviews of complex issues. Furthermore, the National Research Council of the National Academies has recommended that DHS improve its risk analyses for infrastructure protection by validating the models and submitting them to external peer review. As we have previously reported, independent peer reviews cannot ensure the success of a risk assessment approach, but they can increase the probability of success by improving the technical quality of projects and the credibility of the decision-making process. We will continue to monitor and assess ISCD’s efforts to examine its risk assessment approach through our ongoing work and consider any recommendations needed to address these issues.

ISCD Revised Its Security Plan Review Process, but Plan Approvals Could Take Years

ISCD Revised Its Security Plan Review Process because of ISCD Managers’ Concerns, and Plans to Measure Related Improvements Moving Forward

Our preliminary work shows that ISCD has made various revisions to its security plan review process to address concerns expressed by ISCD managers about slow review times. Under the CFATS rule, once a facility is assigned a final tier, it is to submit a site security plan to describe security measures to be taken and how it plans to address applicable risk-based performance standards. The November 2011 internal memorandum that discussed various challenges facing the CFATS program noted that ISCD had not approved any security plans and stated that the process was overly complicated and created bottlenecks. The memorandum stated that revising the process was a top program priority because the initial security plan reviews were conducted using the risk- based standards as prescriptive criteria rather than as standards for developing an overall facility security strategy.

According the ISCD officials, the first revision was called the interim review process, whereby individual reviewers were to consider how layers of security measures met the intent of each of the 18 standards. Under the interim review process, ISCD assigned portions of each facility’s plan to security specialists (e.g., cyber, chemical, and physical, among others) who reviewed plans in a sequential, linear fashion. Using this approach, plans were reviewed by different specialists at different times culminating in a quality review. ISCD officials told us that the interim review process was unsustainable, labor-intensive, and time-consuming, particularly when individual reviewers were looking at pieces of thousands of plans that funneled to one quality reviewer. In July 2012, ISCD stopped using the interim review process and began using the current revised process, which entails using contractors, teams of ISCD employees (physical, cyber, chemical, and policy specialists), and ISCD field office inspectors, who are to review plans simultaneously.

ISCD officials said that they believe the revised process for reviewing security plans is a “quantum leap” forward, but they did not capture data that would enable them to measure how, if at all, the revised process is more efficient (i.e., less time-consuming) than the former processes. They said that, under the revised process, among other things, field inspectors are to work with facilities with the intent of resolving any deficiencies ISCD identifies in their site security plans. They added that this contrasts with past practices whereby ISCD would review the entire plan even when problems were identified early and not return the plan to the facility until the review was complete, resulting in longer reviews. Moving forward, ISCD officials said they intend to measure the time it takes to complete parts of the revised process and have recently implemented a plan to measure various aspects of the process. Specifically, ISCD’s Annual Operating Plan, published in December 2012, lists 63 performance measures designed to look at various aspects of the site security plan review process—from the point the plans are received by ISCD to the point where plans are reviewed and approved. Collecting data to measure performance about various aspects of the security plan review process is a step in the right direction, but it may take time before the process has matured to the point where ISCD is able to establish baselines and assess its progress.

Security Plan Reviews Could Take Years to Complete, but ISCD Is Examining How It Can Accelerate the Review Process

ISCD has taken action to improve its security plan review process, but based on our preliminary analysis, it could take years to review the plans of thousands of facilities that have already been assigned a final tier. ISCD hopes to address this by examining how it can further accelerate the review process. According to ISCD officials, between July 2012 and December 2012, ISCD had approved 18 security plans, with conditions. ISCD officials told us that, moving forward, they anticipate that the revised security plan review process could enable ISCD to approve security plans at a rate of about 30 to 40 a month.

Using ISCD’s estimated approval rate of 30 to 40 plans a month, our preliminary analysis indicates that it could take anywhere from 7 to 9 years to complete reviews and approvals for the approximately 3,120 plans submitted by facilities that have been final-tiered that ISCD has not yet begun to review. Figure 1 shows our estimate of the number of years it could take to approve all of the security plans for the approximately 3,120 facilities that, as of January 2013, had been final- tiered, assuming an approval rate of 30 to 40 plans a month.

It is important to note that our 7- to 9-year preliminary estimate does not include other activities central to the CFATS mission, either related to or aside from the security plan review process. In addition, our estimate does not include developing and implementing the compliance inspection process, which occurs after security plans are approved and is intended to ensure that facilities that are covered by the CFATS rule are compliant with the rule, within the context of the 18 performance standards. According to ISCD officials, they are actively exploring ways to expedite the speed with which the backlog of security plans could be cleared, such as potentially leveraging alternative security programs, reprioritizing resources, and streamlining the inspection and review requirements. ISCD officials added that they plan to complete authorizations inspections and approve security plans for tier 1 facilities by the first quarter of fiscal year 2014 and for tier 2 facilities by the third quarter of fiscal year 2014.

ISCD Has Increased Its Efforts to Communicate and Work with Facilities, but Does Not Solicit Systematic Feedback on Effectiveness of Its Outreach

ISCD’s External Communication Efforts with Facilities Have Increased since 2007, but Selected Trade Associations Had Mixed Views about ISCD Efforts

Our preliminary work shows that ISCD’s efforts to communicate and work with owners and operators to help them enhance security at their facilities have increased since the CFATS program’s inception in 2007, particularly in recent years. Since 2007, ISCD has taken various actions to communicate with facility owners and operators and various stakeholders—including officials representing state and local governments, private industry, and trade associations—to increase awareness about CFATS. From fiscal years 2007 through 2009, most of ISCD’s communication efforts entailed outreach with owners and operators and stakeholders through presentations to familiarize them with CFATS; field visits with federal, state, and local government and private industry officials; and compliance assistance visits at facilities that are intended to assist facilities with compliance or technical issues. By 2010 and in subsequent years, ISCD had revised its outreach efforts to focus on authorization inspections during which inspectors visited facilities to verify that the information in their security plans was accurate and complete, and other outreach activities including stakeholder outreach.

However, analysis of industry trade associations’ responses to questions we sent them about the program shows mixed views about ISCD’s efforts to communicate with owners and operators through ISCD outreach efforts. For example, 3 of the 11 trade associations that responded to our questions indicated that ISCD’s outreach program was effective in general, 3 reported that the effectiveness of ISCD’s outreach was mixed, 4 reported that ISCD’s outreach was not effective, and 1 respondent reported that he did not know.

ISCD Seeks Informal Feedback, but Does Not Solicit Systematic Feedback on the Effectiveness of Its Outreach Efforts

Our preliminary results indicate that ISCD seeks informal feedback on its outreach efforts but does not systematically solicit feedback to assess the effectiveness of outreach activities, and it does not have a mechanism to measure the effectiveness of ISCD’s outreach activities. Trade association officials reported that in general ISCD seeks informal feedback on its outreach efforts and that members provide feedback to ISCD. Association officials further reported that among other things ISCD has encouraged association members to contact local ISCD inspectors and has hosted roundtable discussions and meetings where members of the regulated community provide feedback, suggest improvements, or make proposals regarding aspects of the CFATS program such as site security plans, alternative security programs, and gasoline storage site risks. Furthermore, according to ISCD officials, while feedback is solicited from the regulated community generally on an informal basis, inspectors and other staff involved in ISCD’s outreach activities are not required to solicit feedback during meetings, presentations, and assistance visits, and inspectors are also not required to follow up with the facilities after compliance assistance visits to obtain their views on the effectiveness of the outreach.

ISCD, as part of its annual operating plan, has established a priority for fiscal year 2013 to develop a strategic communications plan intended to address external communication needs including industry outreach. We have previously reported on the benefits of soliciting systematic feedback. Specifically, our prior work on customer service efforts in the government indicates that systematic feedback from those receiving services can provide helpful information as to the kind and quality of services they want and their level of satisfaction with existing services. We will continue to monitor and assess ISCD’s efforts to develop a systematic way to solicit feedback through our ongoing work and consider any recommendations needed to address this issue.

Chairman Shimkus, Ranking Member Tonko, and members of the subcommittee, this completes my prepared statement. I would be happy to respond to any questions you may have at this time.

GAO Contact and Staff Acknowledgments

For information about this statement please contact Stephen L. Caldwell, at (202) 512-9610 or CaldwellS@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals making key contributions included John F. Mortin, Assistant Director; Chuck Bausell; Jose Cardenas; Michele Fejfar; Jeff Jensen; Tracey King; Marvin McGill; Jessica Orr; and Ellen Wolfe.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.