Critical Infrastructure Protection: DHS Efforts to Assess Chemical Security Risk and Gather Feedback on Facility Outreach Can Be Strengthened
Released: 2013-04-05T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: Facilities that produce, store, or use hazardous chemicals could be of interest to terrorists intent on using toxic chemicals to inflict mass casualties in the United States. As required by statute, DHS issued regulations that establish standards for the security of high-risk chemical facilities. DHS established the CFATS program to assess the risk posed by these facilities and inspect them to ensure compliance with DHS standards. ISCD, which manages the program, places high risk facilities in risk-based tiers and is to conduct inspections after it approves facility security plans. A November 2011 ISCD internal memorandum raised concerns about ISCD's ability to fulfill its mission.

GAO assessed the extent to which DHS has (1) assigned chemical facilities to tiers and assessed its approach for doing so, (2) revised its process to review facility security plans, and (3) communicated and worked with owners and operators to improve security. GAO reviewed DHS reports and plans on risk assessments, security plan reviews, and facility outreach and interviewed DHS officials. GAO also received input from 11 trade associations representing chemical facilities, about ISCD outreach. The results of this input are not generalizable but provide insights.

section_title: What GAO Found
paragraphs: Since 2007, the Department of Homeland Security's (DHS) Infrastructure Security Compliance Division (ISCD) has assigned about 3,500 high-risk chemical facilities to risk-based tiers under its Chemical Facility Anti-Terrorism Standards (CFATS) program, but it has not fully assessed its approach for doing so. The approach ISCD used to assess risk and make decisions to place facilities in final tiers does not consider all of the elements of consequence, threat, and vulnerability associated with a terrorist attack involving certain chemicals. For example, the risk assessment approach is based primarily on consequences arising from human casualties, but does not consider economic consequences, as called for by the National Infrastructure Protection Plan (NIPP) and the CFATS regulation, nor does it consider vulnerability, consistent with the NIPP. ISCD has begun to take some actions to examine how its risk assessment approach can be enhanced, including commissioning a panel of experts to assess the current approach, identify strengths and weaknesses, and recommend improvements. ISCD will need to incorporate the various results of these efforts to help them ensure that the revised risk assessment approach includes all elements of risk. After ISCD has incorporated all elements of risk into its assessment approach, an independent peer review would provide better assurance that ISCD can appropriately identify and tier chemical facilities, better inform CFATS planning and resource decisions, and provide the greatest return on investment consistent with the NIPP.

DHS's ISCD has revised its process for reviewing facilities' site security plans--which are to be approved by ISCD before it performs compliance inspections--but it did not track data on the prior process to measure differences. The past process was considered by ISCD to be difficult to implement and caused bottlenecks in approving plans. ISCD views its revised process to be a significant improvement because, among other things, teams of experts review parts of the plans simultaneously rather than sequentially, as occurred in the past. Moving forward ISCD intends to measure the time it takes to complete reviews, but will not be able to do so until the process matures. GAO estimated that it could take another 7 to 9 years before ISCD is able to complete reviews on the approximately 3,120 plans in its queue which means that the CFATS regulatory regime, including compliance inspections, would likely be implemented in 8 to 10 years. ISCD officials said that they are exploring ways to expedite the process such as reprioritizing resources and streamlining inspection requirements.

DHS's ISCD has also taken various actions to work with owners and operators, including increasing the number of visits to facilities to discuss enhancing security plans, but trade associations that responded to GAO's query had mixed views on the effectiveness of ISCD's outreach. ISCD solicits informal feedback from facility owners and operators on its efforts to communicate and work with them, but it does not have an approach for obtaining systematic feedback on its outreach activities. ISCD's ongoing efforts to develop a strategic communication plan may provide opportunities to explore how ISCD can obtain systematic feedback on these activities. A systematic approach for gathering feedback and measuring the results of its outreach efforts could help ISCD focus greater attention on targeting potential problems and areas needing improvement.

section_title: What GAO Recommends
paragraphs: GAO recommends that DHS enhance its risk assessment approach to incorporate all elements of risk, conduct a peer review after doing so, and explore opportunities to gather systematic feedback on facility outreach. DHS concurred with the recommendations.

section_title: Letter
paragraphs: Facilities that produce, use, or store hazardous chemicals could be of  particular interest to terrorists who are intent on using toxic chemicals to  inflict mass casualties in the United States. These chemicals could be  released from a facility to cause harm to surrounding populations, could  be stolen and used as chemical weapons or as their precursors (the  ingredients for making chemical weapons), or stolen and used to build an  improvised explosive device. The Department of Homeland Security  (DHS) appropriations act for fiscal year 2007 required DHS to issue  regulations to establish risk-based performance standards for securing  high-risk chemical facilities. In 2007, DHS established the Chemical  Facility Anti-Terrorism Standards (CFATS) program to assess the risk  posed by chemical facilities, place high-risk facilities in one of four risk- based tiers, require high-risk facilities to develop security plans, review  these plans, and inspect the facilities to ensure compliance with  regulatory requirements. DHS’s National Protection and Programs  Directorate (NPPD) is responsible for these chemical facility security  regulations. Within NPPD, the Office of Infrastructure Protection (IP),  through its Infrastructure Security Compliance Division (ISCD), oversees  the CFATS program.

In 2011, a leaked internal memorandum prompted some Members of  Congress and chemical facility owners and operators to become  concerned about ISCD’s ability to implement and manage a regulatory  regime under the CFATS program. In December 2011, this  memorandum, prepared by the then ISCD Director, was leaked to the  national media, raising concerns about the management of the program.

The memorandum cited an array of challenges that ISCD had  experienced implementing the CFATS program, including an inability to  hire staff with the needed skills, an overly complicated security plan  review process, and a compliance inspection process that had yet to be  developed. In July 2012, we reported that ISCD had efforts under way to  address the problems highlighted in the internal memorandum and had  developed an action plan to track its progress on various human capital,  mission, and administrative issues. We found that ISCD appeared to be  heading in the right direction, but it was too early to tell if individual action  items were having their desired effect because ISCD was in the early  stages of implementing them and had not yet established performance  measures to assess results. We recommended that ISCD explore  opportunities to develop such measures, where practical. ISCD agreed  with our recommendation and in response, developed an operating plan  that includes information on how ISCD plans to measure performance.  We also noted that some of the action items such as developing an  appropriate information technology platform to support inspection  activities would require a longer-term effort by ISCD. You asked us to  follow up on ISCD’s efforts to address various mission issues such as a  security plan review process that, according to ISCD, was overly  complicated and difficult to implement. Specifically, this report discusses  the extent to which DHS has assigned chemical facilities to risk-based tiers and assessed its  approach for doing so,  revised the process used to review security plans, and communicated and worked with facilities to help improve security.

To meet our objectives, we reviewed the CFATS statute and regulation  (or rule), as well as ISCD policies, processes, and procedures that were  in place from CFATS program inception to date. Regarding assigning  chemical facilities to risk-based tiers, we reviewed and analyzed ISCD  documents including the web-based tools used to collect security  information from facilities; the ISCD risk assessment approach used to  determine a facility’s risk, policies and procedures on risk-based tiering,  among others; and data ISCD collects from facilities to make tiering  determinations. We assessed the reliability of the data collected and  found that the data were sufficiently reliable for the purposes of this  report. We compared our analysis against various criteria such as the  CFATS statute and rule; the National Infrastructure Protection Plan  (NIPP), which sets forth the risk management framework for the  protection and resilience of the nation’s critical infrastructure; and risk  modeling best practices as outlined by the National Academy of Sciences  to determine if ISCD’s risk assessment approach comports with these  criteria and if not, where gaps exist. We also reviewed documents  related to ISCD’s ongoing efforts to review its risk assessment approach  including the statement of objectives, task execution plan, and terms of  reference and compared these documents to the criteria for peer review  as laid out by the National Academy of Sciences as well as our prior work  on peer reviews.

Regarding ISCD’s revisions to the security plan review process, we  reviewed documents such as the November 2011 internal memorandum,  DHS’s Risk-Based Performance Standards Guidance, and ISCD security  plan review policies and procedures, among others. To confirm our  understanding of the security plan review process, we also gathered and  analyzed statistics pertinent to the process to determine how many  security plans had been reviewed, authorized, and approved from  program inception to date. We did not include the facility compliance  inspection process (which is based on the results of the approved security  plans) because ISCD began notifying facilities that their security plans  were approved in December 2012.

Regarding communicating and working with facilities to improve security,  we contacted officials representing 15 trade associations with members  regulated by CFATS and who participated in the Chemical Sector  Coordinating Council to get their perspectives on DHS efforts to work with  facility owners and operators. Out of the 15 trade associations we  contacted, 11 responded and the information we obtained from them is  not generalizable to the universe of chemical facilities covered by CFATS.  However, the information we obtained from them provides insights into  DHS efforts to perform outreach and seek feedback on the  implementation of the CFATS rule. For all our objectives, we interviewed  ISCD officials responsible for overseeing the CFATS program to confirm  our understanding of the documents and data provided. Appendix I  discusses our scope and methodology in greater detail.

We conducted this performance audit from October 2012 through April  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The CFATS program is intended to secure the nation’s chemical  infrastructure by identifying and protecting high-risk chemical facilities.  Section 550 of the DHS appropriations act for fiscal year 2007 requires  DHS to issue regulations establishing risk-based performance standards  for the security of facilities that the Secretary determines to present high  levels of security risk. The CFATS rule was published in April 2007  and Appendix A to the rule, published in November 2007, listed 322  chemicals of interest and the screening threshold quantities amount for  each.

ISCD has direct responsibility for implementing DHS’s CFATS rule,  including assessing risks and identifying high-risk chemical facilities,  promoting effective security planning, and ensuring that high-risk facilities  meet the applicable risk-based performance standards through site  security plans approved by DHS. ISCD is managed by a Director and  operates five branches that are, among other things, responsible for (1)  information technology operations; (2) policy and planning; (3) providing  compliance and technical support; (4) inspecting facilities and enforcing  CFATS regulatory standards; and (5) managing logistics, administration,  and chemical security training. From fiscal years 2007 through 2012, DHS dedicated about $442 million to the CFATS program. In fiscal year  2012, DHS was authorized 242 full-time equivalent positions.
subsections: section_title: The CFATS Rule and Process
paragraphs: DHS’s CFATS rule outlines a specific process for administering the  program. Any chemical facility that possesses any of the 322 chemicals in  the quantities that meet or exceed the threshold quantity outlined in  Appendix A of the rule is required to use DHS’s Chemical Security  Assessment Tool (CSAT)—a web-based application through which  owners and operators of chemical facilities provide information about the  facility. Once a facility is registered in CSAT, owners and operators are  to complete the CSAT Top Screen—which is the initial screening tool or  document whereby the facility is to provide DHS various data, including  the name and location of the facility and the chemicals and their  quantities at the site. DHS is to analyze this information using its risk  assessment approach, which is discussed in more detail below, to initially  determine whether the facility is high risk. If so, DHS is to notify the  facility of its preliminary placement in one of four risk-based tiers—tier 1,  2, 3, or 4. Facilities preliminarily placed in any one of these tiers are  considered to be high risk, with tier 1 facilities considered to be the  highest risk. Facilities that DHS initially determines to be high risk are  required to then complete the CSAT security vulnerability assessment,  which includes the identification of potential critical assets at the facility  and a related vulnerability analysis. DHS is to review the security  vulnerability assessment and notify the facility of DHS’s final  determination as to whether or not the facility is considered high risk, and  if the facility is determined to be a high-risk facility, about its final  placement in one of the four tiers.

Once assigned a final tier, the facility is required to use CSAT to submit a  site security plan or participate in an alternative security program in lieu of  a site security plan. The security plan is to describe the security  measures to be taken to address the vulnerabilities identified in the  vulnerability assessment, and identify and describe how security  measures selected by the facility are to address the applicable risk-based  performance standards. DHS then is to conduct a preliminary review of  the security plan to determine whether it meets the regulatory  requirements. If these requirements appear to be satisfied, DHS is to  issue a letter of authorization for the facility’s plan. DHS then is to conduct  an authorization inspection of the facility and subsequently determine  whether to approve the security plan. If DHS determines that the plan  does not satisfy CFATS requirements, DHS then notifies the facility of any  deficiencies and the facility must submit a revised plan correcting them.  If the facility fails to correct the deficiencies, DHS may disapprove the  plan. Following approval, DHS may conduct further inspections to  determine if the facility is in compliance with its approved security plan.Figure 1 illustrates the CFATS regulatory process.

ISCD uses a risk assessment approach during the early stages of the  regulatory process to develop risk scores to assign chemical facilities to a  final tier. According to an ISCD document that describes how ISCD  develops its CFATS risk score, the risk score is intended to be derived  from estimates of consequence (the adverse effects of a successful  attack), threat (the likelihood of an attack), and vulnerability (the likelihood  of a successful attack, given an attempt). The ISCD risk assessment  approach is composed of three models, each based on a particular  security issue: (1) release, (2) theft or diversion, and (3) sabotage,  depending on the type of risk associated with the 322 chemicals of  interest listed in Appendix A of the CFATS rule. For release, the model  assumes that a terrorist will release the chemical of interest at the facility  and then estimates the risk to the surrounding population. For theft or  diversion, the model assumes that a terrorist will steal or have the  chemical of interest diverted to him or herself and then estimates the risk  of a terrorist attack using the chemical of interest in a way that causes the  most harm at an unspecified off-site location. For sabotage, the model  assumes that a terrorist will remove the chemical of interest from the  facility and mix it with water, creating a toxic release at an unspecified off- site location, and then estimates the risk to a medium-sized U.S. city.  Once ISCD estimates a risk score based on these models, it assigns the  facility to a final tier.

section_title: ISCD Has Assigned Thousands of Facilities to Tiers, but ISCD’s Approach to Risk Assessment Does Not Reflect All Risk Elements
paragraphs: Since 2007, ISCD has assigned about 3,500 high-risk chemical facilities  to final tiers and has taken action to identify and address problems with its  risk-tiering approach. However, ISCD’s risk-tiering approach does not  reflect all elements of risk. Specifically, ISCD is to assess risk using  estimates of the consequences, threat, and vulnerability associated with a  terrorist attack, but ISCD does not consider key elements of risk, such as  economic consequences or facility vulnerability consistent with the NIPP  and the CFATS rule. ISCD recognizes that its tiering approach is not  complete and continues to mature and has begun to take actions to  assess its approach, including commissioning an expert panel.
subsections: section_title: ISCD Has Tiered Thousands of High-Risk Chemical Facilities and Resolved Some Problems Using Its Risk Assessment Approach to Assign Tiers
paragraphs: In July 2007, ISCD began reviewing information submitted by the owners  and operators of approximately 40,000 facilities. By January 2013, ISCD  had designated about 4,400 of the 40,000 facilities as high risk and  thereby covered by the CFATS rule. ISCD had assigned about 3,500 of  those facilities to a final tier, of which about 90 percent were tiered  because of the risk of theft or diversion. The remaining 10 percent were  tiered because of the risk of release or the risk of sabotage. In addition,  about 900 of the 4,400 facilities had been assigned to preliminary tiers  and were to be assigned a final tier once ISCD processed data from the  facility using ISCD’s risk assessment approach. ISCD officials noted that  the number of tiered facilities and their individual tiers is likely to be fluid  over time as changes in chemical holdings, production, processes,  storage methods, or use occur. Table 1 shows the number and  percentage of facilities assigned a final tier as of January 2013.

Over the last 2 years, ISCD has identified problems with the way the  release chemicals model assigns chemical facilities to tiers and has taken  or begun to take action to address those problems. In February 2011,  ISCD managers were notified by contracting officials responsible for  running the model that some chemical facilities had been placed in an  incorrect final tier because this model included incorrect data about the  release of high-risk chemicals of interest. In June 2011, ISCD officials  adjusted the model, lowering the tier for about 250 facilities, about 100 of  which were subsequently removed from the CFATS program. In  September 2012, ISCD officials stated that they were confident that the  adjustment helped make this model more accurate.

However, in October 2012, ISCD officials stated that they had discovered  another anomaly that they were working to correct. Specifically, ISCD  officials said that they had uncovered a defect that led the model to  exclude population density calculations for about 150 facilities in states or  U.S. territories outside the continental United States, including Alaska,  Hawaii, Puerto Rico, and Guam. In December 2012, ISCD officials said  that they had made adjustments to the model to resolve this issue. They  added that they expected that once data from the approximately 150  facilities were assessed, no more than 11 of the approximately 150  facilities would be affected by a change to their tier. ISCD officials said  that as of February 2013, upon further examination, they expect that  about 2 facilities will be affected. However, those two facilities were  already tiered for other chemicals covered by CFATS, and ISCD officials  did not expect those facilities’ respective tiers to change.

section_title: ISCD’s Risk Assessment Approach Does Not Consider All Elements of Risk
paragraphs: ISCD has tiered thousands of facilities using its current risk assessment  approach, but ISCD’s risk assessment approach is not mature because it  does not consider key elements of risk from the NIPP and the CFATS  rule. According to the NIPP, which, among other things, establishes the  framework for managing risk among the nation’s critical infrastructure, risk  is a function of three components—consequence, threat, and  vulnerability—and a risk assessment approach must assess each  component for every defined risk scenario. Furthermore, the CFATS rule  calls for ISCD to review consequence, threat, and vulnerability  information in determining a facility’s final tier. However, ISCD’s risk  assessment approach does not fully consider all of the core criteria or  components of a risk assessment, as specified by the NIPP, nor does it  comport with parts of the CFATS rule.

ISCD’s risk assessment approach does not currently conform to the NIPP  and is not consistent with the CFATS rule because it does not yet fully  consider consequence criteria when assessing risk associated with a  terrorist attack. The NIPP states that at a minimum, consequences should  focus on the two most fundamental components—human consequences  and the most relevant direct economic consequences. Like the NIPP, the  CFATS rule states that chemical facilities covered by the rule are those  that present a high risk of significant adverse consequences for human  life or health, or critical economic assets, among other things, if subjected  to terrorist attack, compromise, infiltration, or exploitation.

Our review of ISCD’s risk assessment approach and discussions with  ISCD officials showed that the approach is currently limited to focusing on  one component of consequences—human casualties associated with a  terrorist attack involving a chemical of interest—and does not consider  consequences associated with economic criticality. ISCD officials told us  that, at the inception of the CFATS program, they did not have the  capability to collect or process all of the economic data needed to  calculate the associated risks and they were not positioned to gather all of  the data needed. They said that they collect basic economic data as part  of the initial screening process in the CSAT; however, they would need to  modify the current tool to collect more sufficient data. This contrasts with  other DHS components, like the U.S. Coast Guard and the Transportation  Security Administration, which have gathered and assessed economic  data as part of some critical infrastructure risk assessment efforts.

ISCD officials stated that they have begun to have discussions with other  DHS components, like the U.S Coast Guard, about their approach to risk  assessment. They also said that they recognize that the economic  consequences part of their risk-tiering approach will require additional  work before it is ready to be introduced. They noted that the preamble to  the November 2007 CFATS rule stated that they would defer  incorporating economic criticality until a later date. In September 2012,  ISCD officials told us that they had engaged Sandia National Laboratories  to examine how ISCD could gather needed information and determine the  risk associated with economic impact, but this effort is in the initial stages,  with an expected completion date of June 2014. ISCD officials added  they are uncertain about how Sandia National Laboratories’ efforts will  affect their risk assessment approach.

ISCD’s risk assessment approach is also not consistent with the NIPP  because it does not consider threat for the majority of regulated facilities.  According to the NIPP, risk assessments should estimate threat as the  likelihood that the adversary would attempt a given attack method against  the target. Like the NIPP, the CFATS rule requires that, as part of site  vulnerability assessment process, facilities conduct a threat assessment,  which is to include a description of the internal, external, and internally- assisted threats facing the facility and that ISCD review the site  vulnerability assessment as part of the final determination of a facility’s  tier. showed that (1) ISCD is inconsistent in how it assesses threat using the  different models because while it considers threat for the 10 percent of  facilities tiered because of the risk of release or sabotage, it does not  consider threat for the approximately 90 percent of facilities that are tiered  because of the risk of theft or diversion; and (2) ISCD does not use  current threat data for the 10 percent of facilities tiered because of the risk  of release or sabotage.

6 C.F.R. §§ 27.215, .220.

ISCD did not have documentation to show why threat had not been  factored into the formula for approximately 90 percent of facilities tiered  because of the risk of theft or diversion. However, they pointed out that  the cost of adding a threat analysis for these facilities might outweigh the  benefits of doing so because it may not provide the increased specificity  and level of details to justify the cost. Officials further explained that the  model assumes that a terrorist would remove the chemical of interest and  use it offsite and ISCD cannot predict where a chemical of interest would  be used as a result of theft or diversion. Nonetheless, it is inconsistent for  ISCD to not consider threat for the theft or diversion risk model, given that  the assumptions about an attack are similar to those considered under  the sabotage model—that is, both models assume that a terrorist would  use a chemical of interest at an offsite, undisclosed location. This extra  level of specificity would be useful for ISCD’s overall risk assessment  efforts given that about 90 percent of facilities are regulated because of  the theft or diversion security issue. ISCD officials said that given the  complexity of assessing threat for theft or diversion, they are considering  reexamining their approach.

Regarding the other 10 percent—facilities tiered because of the risk of  release or sabotage—ISCD documents showed that both models  consider threat data based primarily on the location of the facility.  Nonetheless, ISCD could use more current data to estimate threat among  these facilities. Our review showed that ISCD is using 5-year-old threat  data based on metropolitan statistical areas (MSA) to estimate threat for  those facilities even though these data are updated annually by DHS for  purposes of the Urban Areas Security Initiative program. ISCD officials  said that they were unaware that threat data they were using were out of  date and said they would explore the feasibility of using updated threat  scores. Current threat data would provide a more complete and accurate  threat profile for release or sabotage and might aid in ISCD’s overall risk  assessment efforts.

ISCD’s risk assessment approach is also not consistent with the NIPP  because it does not consider vulnerability when developing risk scores.

According to the NIPP, risk assessments should identify vulnerabilities,  describe all protective measures, and estimate the likelihood of an  adversary’s success for each attack scenario. Similar to the NIPP, the  CFATS rule calls for ISCD to review facilities security vulnerability  assessments as part of its risk-based tiering process. This assessment  is to include the identification of potential security vulnerabilities and the  identification of existing countermeasures and their level of effectiveness  in both reducing identified vulnerabilities and meeting the aforementioned  risk-based performances standards.

Our review of the risk assessment approach and discussions with ISCD  officials showed that the security vulnerability assessment—the primary  CSAT application ISCD uses to assess risk—contains numerous  questions aimed at assessing vulnerability and security measures in  place. These include questions about the accessibility of the facility to an  attacker, the capability of the security force to respond to an attack, and  security controls related to potential cyber attacks. However, although  facilities are required to respond to these questions, ISCD officials told us  that they have opted not to use the data facilities provide because it is  “self-reported” data—data that are not validated by ISCD—and ISCD  officials have observed that facility owners and operators tend to either  overstate or understate some of the vulnerability information provided;  thus making it not useful for tiering purposes. ISCD officials agreed that  the risk assessment approach does not assess differences in vulnerability  from facility to facility and location to location because it does not use any  vulnerability data. Thus, ISCD’s risk assessment approach treats every  facility as equally vulnerable to a terrorist attack regardless of location  and on-site security.

ISCD officials told us that they consider facility vulnerability, but primarily  at the latter stages of the CFATS regulatory process particularly with  regard to the development and approval of the facility site security plan  and the inspection process. With regard to site security plans, ISCD  officials stated that even though facility data are not currently used to tier  facilities based on their response in the security vulnerability assessment,  they view the responses as valuable because they prompt facilities’  thinking about vulnerability before they prepare their site security plan or  alternative security program. Regarding inspections, ISCD officials stated  that they believe that once security plans are authorized and approved,  the inspection process could enable ISCD to assess facilities’  vulnerabilities and gauge their progress mitigating those vulnerabilities.  Because ISCD has completed a limited number of authorization  inspections (56 as of December 2012), it is too early to tell how they plan  to use this self-reported vulnerability information. However, ISCD officials  indicated that it might be used to help make decisions about the use of  inspection resources, especially since they do not anticipate retiering  facilities based on their efforts to mitigate risk.

section_title: ISCD Has Begun to Take Actions to Examine How its Approach Could be Enhanced and Could Take Additional Steps to Help Ensure That it is Complete and Validated
paragraphs: ISCD has begun to take some actions to examine how its risk  assessment approach can be enhanced. For example, in addition to  engaging Sandia National Laboratories to develop the framework for  assessing economic consequences discussed earlier, ISCD has  commissioned a panel of subject matter experts to examine the strengths  and weaknesses, if any, of its current risk assessment approach. ISCD  officials stated that the panel’s work is intended to focus on whether ISCD  is heading in the right direction and they view it as a preliminary  assessment. According to ISCD’s task execution plan, the objectives of  this assessment are to (1) convene a panel of subject matter experts  involved in chemical safety and security, (2) hold one or more working  group meetings focused on assessing and providing feedback on the  current models and (3) provide a report on the strengths, weaknesses,  and issues on the current models. The plan calls for the panel to provide  actionable recommendations on potential improvements to the CFATS  models, but the panel is not to develop alternative CFATS models nor  formally validate or verify the current CFATS risk assessment approach— steps that would analyze the structure of the models and determine  whether they calculate values correctly. ISCD officials stated that they  believe that the review process would include some steps to assess  whether the models are methodologically sound and reliable. In February  2013, after the panel was convened, ISCD officials also stated that they  provided information to the panel about various issues that they might  want to consider, among them (1) how to address vulnerability in the  models given ISCD concerns about data quality and (2) what the  appropriate variables to use, if any, are for threats associated with theft or  diversion, as discussed earlier.

ISCD is moving in the right direction by commissioning the panel to  identify the strengths and weaknesses of its risk assessment approach  and the results of the panel’s work could help ISCD identify issues for  further review and recommendations for improvement. The results of the  panel’s efforts represent one piece of information ISCD will have to  consider, moving forward, to ensure that the risk assessment approach is  complete within the context of the NIPP risk management framework and  the CFATS rule. For instance, in addition to any recommendations  coming out of the panel’s work, the development of a mature risk  assessment approach would require that ISCD consider and act upon the  results of Sandia National Laboratories work on economic consequences.  Likewise, ISCD would need to consider the issues we identified, such as  not using up-to-date threat data, or how vulnerability could be used in the  final tiering process.

ISCD will need to develop an overall plan designed to incorporate the  results of these various efforts to revise and enhance its risk assessment  approach to fully address each of the components of risk— consequences, threat, and vulnerability—to better align them with the  NIPP and the CFATS rule. A plan, complete with milestones and time  frames, is consistent with standard practices for project management,  which state that managing a project involves, among other things,  developing a timeline with milestone dates to identify points throughout  the project to reassess efforts under way to determine whether project  changes are necessary. ISCD would then be better situated to provide a  more complete picture of its approach for developing and completing its  review of steps needed to address each component of ISCD’s risk  assessment approach and actions needed to make it fully conform to the  NIPP and the CFATS rule. It also would provide ISCD managers and  other decision makers with insights into (1) ISCD’s overall progress and  (2) a basis for determining what, if any, additional actions need to be  taken.

In addition, given the significant consequences of a terrorist attack on a  chemical facility, after ISCD completes these actions, commissioning an  independent peer review to assess its revised risk assessment approach,  including a complete verification and validation of the models would help  ensure that the revised model is sound and facilities are appropriately  tiered. In our past work, we reported that peer reviews are a best practice  in risk management and that independent expert review panels can  provide objective reviews of complex issues. We reported that peer  reviews should, among other things, address the structure of the model,  the types and certainty of the data, and how the model is intended to be  used. Furthermore, the National Research Council of the National  Academies has recommended that DHS improve its risk analyses for  infrastructure protection by validating the models and submitting them to  external peer review. According to the National Research Council of the  National Academies, peer reviews should include validation and  verification to ensure that the structure of the models is both accurate and  reliable.

As we have previously reported, independent peer reviews cannot ensure  the success of a risk assessment approach, but they can increase the  probability of success by improving the technical quality of projects and  the credibility of the decision-making process. Thus, a peer review that is commissioned after ISCD revises its approach and incorporates all of  the elements of risk would enable peer reviewers to consider a more  complete risk assessment approach and provide the opportunity to fully  verify and validate it. After ISCD has developed a more mature risk  assessment approach, a subsequent peer review would provide better  assurance that ISCD can appropriately identify and tier chemical facilities,  better inform CFATS planning and resource decisions; and provide the  greatest return on investment consistent with the NIPP.

section_title: ISCD Revised Its Security Plan Review Process, but Plan Approvals Could Take Years
paragraphs: ISCD has revised its site security plan review process to address  concerns expressed by ISCD managers that the original process was  overly complicated and included bottlenecks that slowed the review time.  ISCD officials said that they believe the current security plan review  process, implemented in July 2012, is an improvement over the prior  versions. However, they did not collect or track data on the prior review  processes, so the improvement between the previous review processes  and the current process cannot be measured. Going forward, ISCD has  recently implemented a plan to measure various aspects of the process,  but it will take time before ISCD can establish baseline measures.  Nonetheless, given the rate at which ISCD intends to review and approve  security plans, we estimate that it could take about 7 to 9 years to  complete reviews of plans for approximately 3,120 facilities that, as of  January 2013, had been assigned a final tier but had not yet had their  security plans reviewed and approved.
subsections: section_title: ISCD Revised Its Security Plan Review Process because of ISCD Managers’ Concerns
paragraphs: ISCD has made various revisions to its security plan review process.  Under the CFATS rule, once a facility is assigned a final tier, it is to  submit a site security plan to describe security measures to be taken and  how they will address applicable risk-based performance standards.  The November 2011 internal memorandum that discussed various  challenges facing the program noted that ISCD had not approved any  security plans and stated that the process was overly complicated, did not  leverage ISCD’s resources, and created bottlenecks. In addition, the  memorandum stated that revising the process was a top program priority  because the initial security plan reviews were conducted in a manner  inconsistent with the spirit and intent of the CFATS authorizing  legislation—that is, plan reviewers used the risk-based standards as  prescriptive criteria rather than as standards for developing an overall  facility security strategy. According to ISCD, the initial reviews were  conducted using the 18 risk-based standards as prescriptive criteria  because ISCD had not developed guidance for reviewers of facility plans  to use when considering the merits of those plans. ISCD officials told us  that they had been working on a solution prior to the internal  memorandum being finalized in November 2011. They also pointed out  that the action plan that was intended to address the challenges outlined  in the memorandum, developed in early 2012, included an action item  devoted to improving the security plan review process.

ISCD has implemented two revisions to the security plan review process  since October 2011. According the ISCD officials, the first revision was  called the interim review process, which was intended to be a “holistic”  review whereby individual reviewers were to consider how layers of  security measures met the intent of each of the 18 standards. This was a  departure from the original review process which generally used the  performance standards as specific criteria. Under the interim process,  ISCD assigned portions of each facility’s plan to security specialists (e.g.,  cyber, chemical, and physical, among others) who reviewed plans in a  sequential, linear fashion. Using this approach, plans were reviewed by  different specialists at different times culminating in a quality review. ISCD  officials told us that the interim process was unsustainable, labor- intensive, and time-consuming, particularly when individual reviewers  were looking at pieces of thousands of plans that funneled to one quality  reviewer.

In July 2012, ISCD stopped using the interim process and began using  the revised review process. The current process entails using contractors,  teams of ISCD employees (physical, cyber, chemical, and policy  specialists), and ISCD field office inspectors who are to review plans  simultaneously using the holistic approach developed earlier. Figure 2  shows the revised security plan review process as of July 2012.

ISCD officials said that they believe the revised security plan review  process is a “quantum leap” forward, but did not capture data that would  enable them to measure how, if at all, the new process is more efficient  (i.e., less time-consuming) than the former processes. ISCD officials  explained that one of the more time-saving beneficial aspects of the new  process involves field inspectors interacting with the facilities when the  review of the security plan results in an unfavorable outcome. Now, when  ISCD identifies a security plan that contains deficiencies, such as missing  or unclear information about a security measure, the plan is to be  immediately returned to the facility and ISCD is to schedule a compliance  assistance visit whereby field inspectors work with the facility to resolve  any issues identified. According to ISCD officials, this approach contrasts  with the past practices whereby ISCD would continue to review the entire  plan even when problems were identified early and not return the plan to  the facility until the review was complete, resulting in longer reviews.

Officials also noted that by using the revised process, ISCD has realized  the value of (1) moving from a single person reviewing every plan  sequentially to a team approach, and (2) understanding that security  plans do not have to be perfect in order to issue authorization letters and  conduct authorization inspections. Regarding the latter, ISCD officials  noted that ISCD has begun issuing authorization letters with conditions to  inform facilities that their plans provide sufficient information to schedule  an inspection. For example, one authorization letter noted that ISCD had  not yet determined whether or not the plan satisfied the cyber security  risk-based performance standard, and stated that additional information  would be gathered during the authorization inspection.

Also, when the revised process was implemented in July 2012, all  authorization letters include a condition noting that ISCD has not fully  approved the personnel surety risk-based performance standard of plans  because ISCD has not yet determined what the facilities are to do to meet  all aspects of personnel surety. ISCD believes issuing authorization  letters with conditions, rather than waiting until all conditions are met,  enables inspectors to visit facilities sooner so that ISCD can approve  plans more quickly.

Moving forward, ISCD intends to measure the time it takes to complete  parts of the new process and has recently implemented a plan to  measure various aspects of the process. Specifically, ISCD’s Annual  Operating Plan, published in December 2012, lists 63 performance  measures designed to look at various aspects of the site security plan  review process—from the point the plans are received by ISCD to the  point where plans are reviewed and approved. For example, ISCD plans  to collect data on (1) the percentage of facilities with authorization  inspections completed within 90 days of security plan authorization for tier  1 and 2 facilities and within 120 days of security plan authorization for tier  3 and 4 facilities, and (2) the number of high-risk facilities in total and by  tier that have approved security plans. Collecting data to measure  performance about various aspects of the security plan review process is  a step in the right direction, but it may take time before the process has  matured to the point where ISCD is able to establish baselines and  assess its progress. As of February 2013, ISCD is beginning to gather  data at points in the process to establish baselines and measure  performance and has a goal of reviewing some of the measures and data  associated with them monthly, quarterly, or annually, depending on the  measure.

section_title: Security Plan Reviews Could Take Years to Complete, but ISCD is Examining How it Can Accelerate the Review Process
paragraphs: ISCD actions to revise its security plan reviews may result in  improvements over the prior processes, but it could take years to review  plans for thousands of facilities that have been assigned a final tier—a  factor which ISCD hopes to address by examining how it can accelerate  the review process. According to ISCD officials, between July 2012 and  December 2012, ISCD had approved 18 security plans, with conditions  such as the aforementioned personnel surety qualification. ISCD officials  told us that, moving forward, they anticipate that the revised security plan  review process could enable ISCD to approve security plans at a rate of  about 30 to 40 a month. Furthermore, ISCD officials noted that the  approval rate could reach 50 plans a month in the third quarter of fiscal  year 2013 as the review process becomes more efficient. However, ISCD  estimates that under a best case scenario the revised review process  could take about 6 months to approve a plan—assuming the plan would  not have to be sent back to the facility for revision. ISCD estimates further  show that under a worst case scenario the revised process could take as  long as 20 months to approve a plan—assuming the plan would have to  be sent back to the facility for revisions. Regardless, ISCD officials told us  that they would likely be able to increase production because staff are  overcoming the learning curve associated with the revised process.

Using ISCD’s estimated approval rate of 30 to 40 plans a month, we  calculated that it could take anywhere from 7 to 9 years to complete  reviews and approvals for the approximately 3,120 plans submitted by  facilities that have been final-tiered that ISCD has not yet begun to  review. Figure 3 shows the estimated number of years it could take to  approve all of the security plans for the approximately 3,120 facilities that,  as of January 2013, had been final-tiered assuming an approval rate of  30 to 40 plans a month.

It is important to note that our 7 to 9 year estimate does not include other  activities central to the CFATS mission, either related to or aside from the  security plan review process. Specifically, our estimate does not include  time required to:  review security plans for about 900 facilities that have yet to be  assigned a final tier; and review approved security plans to resolve issues relating to personnel  surety, which cannot be fully accomplished until after ISCD decides  how to conduct the terrorist ties portion of personnel surety.

Finally, our estimate does not include developing and implementing the  compliance inspection process, which is intended to ensure that facilities  that are covered by the CFATS rule are compliant with the rule, within the  context of the 18 performance standards. ISCD officials estimate that the  first compliance inspections would commence in September 2013, which  means that the CFATS regulatory regime would likely be fully  implemented for currently tiered facilities (to include compliance  inspections) in 8 to 10 years. According to ISCD officials, they are actively  exploring ways to expedite the speed with which the backlog of security  plans will be cleared such as potentially leveraging alternative security  programs, re-prioritizing resources, and streamlining the inspection and  review requirements. ISCD officials added that they plan to complete  authorizations inspections and approve security plans for tier 1 facilities  by the first quarter of fiscal year 2014 and for tier 2 facilities by the third  quarter of fiscal year 2014.

section_title: ISCD Has Increased its Efforts to Communicate and Work with Facilities and May Have an Opportunity to Systematically Gather Feedback on Its Outreach Efforts
paragraphs: ISCD’s efforts to communicate and work with owners and operators to  help them enhance security at their facilities have increased since the  CFATS program’s inception in 2007, particularly in recent years. The  various trade associations representing facility owners and operators who  responded to our query on ISCD’s outreach had mixed views about the  effectiveness of ISCD’s efforts to communicate with them over various  aspects of the program. Most of the trade associations that responded  stated that ISCD seeks and receives informal feedback on its  communication efforts, but ISCD stated that it has not developed a  systematic approach to solicit feedback to assess the effectiveness of its  outreach activities. ISCD is currently developing a strategic  communication plan which may create an opportunity for ISCD to explore  how it can obtain systematic feedback on its outreach.
subsections: section_title: ISCD’s External Communication Efforts with Facilities have Increased Since 2007
paragraphs: Since 2007, ISCD has taken various actions to communicate with facility  owners and operators and various stakeholders—including officials  representing state and local governments, private industry, and trade  associations—to increase awareness about CFATS and these efforts  have increased as the program has matured. From fiscal years 2007  through 2009, most of ISCD’s communication efforts entailed outreach  with owners and operators and stakeholders through presentations to  familiarize them with CFATS; field visits with federal, state, and local  government and private industry officials; and compliance assistance  visits at facilities that are intended to assist them with compliance or  technical issues. By 2010 and in subsequent years, ISCD revised its  outreach efforts to focus on authorization inspections during which  inspectors visited facilities to verify that the information in their security  plans was accurate and complete and other outreach activities including  stakeholder outreach. Table 2 shows the number of outreach activities  performed by ISCD from fiscal year 2007 through the first quarter of fiscal  year 2013.

According to ISCD officials, the increase in outreach was intended to  facilitate the development of site security plans and occurred for various  reasons. First, according to ISCD officials, during the early years of the  program, regulated facilities did not require as much assistance because  they were generally engaged in the development of their Top Screens  and security vulnerability assessments. Second, officials said that, as  ISCD matured, its ability to track outreach activities improved when  ISCD’s tracking system was automated in June 2010—during the early  years of the program, outreach reporting was manual. Third, ISCD  officials stated that ISCD staffing increases made it possible to perform  more outreach to regulated stakeholders. They said that, prior to fiscal  year 2009, field staff consisted of about 30 staff detailed from DHS’s  Federal Protective Service and in subsequent years ISCD increased the  size of its field staff to more than 100.

section_title: Selected Trade Associations Had Mixed Views about ISCD Efforts to Communicate with Owners and Operators
paragraphs: Our analysis of industry trade associations’ responses to questions we  sent them about the program showed mixed views about ISCD’s efforts to  communicate with owners and operators through ISCD outreach efforts.  Whereas 3 of the 11 trade associations that responded to our questions  indicated that ISCD’s outreach program was effective in general, 3  reported that the effectiveness of ISCD’s outreach was mixed, 4 reported  that ISCD’s outreach was not effective, and 1 respondent reported that he  did not know. Our analysis also showed that trade associations that  responded, in general, viewed specific types of ISCD outreach to be  either effective or of mixed effectiveness (fig. 4 shows our analysis of  trade association responses to questions about specific types of ISCD’s  outreach activities).

We also analyzed trade association responses with regard to the  usefulness of ISCD outreach activity in terms of their members’  understanding of performance standards, tiering approach, and data  collection requirements, as shown in figure 5.

Our analysis of trade association responses to our questions showed that  opinions were mixed regarding the usefulness of outreach activities  related to helping regulated facilities understand the Risk-Based  Performance Standards. Specifically, trade association responses  showed that 5 of the 11 associations that responded indicated that  outreach was useful or had mixed usefulness and 4 associations reported  that it was not useful. For example, among those trade associations that  indicated that outreach on the performance standards was useful, one  reported that outreach activities have helped members better understand  the standards and another reported that performance standards guidance  and early presentations were also helpful. Conversely, among those that  indicated they did not believe outreach on the Risk-Based Performance  Standards was useful, one cited a lack of ISCD training on the standards  and another reported that ISCD needed to provide more assistance to  explain what would or would not be compliant using the standards.

Nearly all (9 of 11) trade association respondents indicated that ISCD’s  outreach to members was not useful in helping them understand the  tiering approach used to determine the risk levels of regulated facilities,  which they viewed as not being transparent. respondents reported that the lack of transparency can hinder facility  owners and operators’ ability to properly make risk reduction decisions;  and 6 respondents reported that there are instances where tiering results  contradict the facility’s perception of what is “high risk” or conflict with  results at similar facilities with similar chemical holdings and population  densities in surrounding communities. One member company of 1 trade  association reported that industry has repeatedly asked for information on  the tiering process without success. ISCD officials told us that they do not  currently provide regulated facilities with details about the CFATS risk- tiering approach but noted that one of the tasks of the aforementioned  expert panel review is to make recommendations regarding what  additional tiering-related information should be provided to facilities. DHS  has designated some parts of the risk models as Chemical-terrorism  Vulnerability Information—which warrants special treatment for handling  and sharing, including assessing whether or not there is a need to  know—and other parts of the models as Secret. ISCD officials also  stated that they have not received a formal request by facilities asking  ISCD to recheck assigned tiering levels or to re-evaluate the output of the  risk models because facilities believe the methodology used is faulty.

One respondent said that he was not well positioned to comment on the usefulness of  ISCD’s tiering-related outreach activities because of a lack of information from owners and  operators the association represented. Another respondent indicated that that the  association was not aware that ISCD had conducted outreach in this area.

According to these officials, the primary request from facilities is to know  how their facilities were tiered, and industry requests for information  mainly come from facilities that either believe they should not be tiered or  tiered facilities that are questioning why a neighboring facility has not  been tiered.

Our query of selected trade associations also showed that they generally  had mixed views about ISCD’s outreach on data collection requirements.  Specifically, 4 of the 11 association officials that provided responses to  our query reported that ISCD’s outreach efforts had either been useful or  had been of mixed usefulness in enabling them to better understand and  comply with the ISCD’s data collection requirements; 3 reported that  outreach on data collection requirements had not been useful; and 4 did  not answer the question. Among the 4 respondents that found outreach  on data collection requirements to be useful or of mixed usefulness, 1  noted that ISCD had done more outreach specifically on elements of the  site security plan, which has broadened stakeholders understanding of  the type of information DHS is looking for and options they may not have  previously considered. Of the 3 that reported ISCD’s outreach on data  collection requirements had not been useful, 1 trade association  respondent reported that, in 2009, industry representatives had  suggested that ISCD produce a document providing tips and suggestions  for completing site security plans and reported that it took 3 years for  ISCD to produce such a document; similarly, 2 member companies of  another trade association reported that ISCD’s outreach had not been  useful in increasing understanding of data collection requirements, and 1  cited the experience of an individual that seemed confused on much of  the information being requested.

section_title: Trade Associations We Contacted Expressed Concern about the Burden of Responding to Data Collection Requirements which ISCD Has Plans to Address
paragraphs: Most of the trade associations that responded to our questions also  expressed concern about CSAT data collection requirements specifically  with regard to various applications such as the Top Screen, vulnerability  assessment, and site security plan. Specifically, our analysis of  responses by the 11 trade associations indicated that nearly all (9 of 11  respondents) believed that the CSAT data collection effort was  burdensome for regulated owners and operators; the other 2 trade  associations provided mixed responses. Nine industry trade associations  reported that the CSAT data submission requirements take significant  resources (such as time and personnel) to prepare, and 10 questioned  the value of the tools in reducing risk or increasing security.association official reported that completing the Top Screen, vulnerability  assessment, and security plan data collection effort required over 200  person hours. Another association official reported that the vulnerability  assessment data collection requirements were very burdensome and  noted that one member’s security plan covered 1,400 questions and was  nearly 300 pages long.

ISCD officials acknowledged that CSAT can be burdensome and they  intend to introduce improvements to CSAT to assist facilities in  developing and submitting their Top Screens, security vulnerability  assessments, and site security plans. In ISCD’s December 2012 Annual  Operating Plan, one action item calls for the revision of the CSAT based  on engagement with industry in order to create a more efficient and  effective tool. ISCD officials stated that they estimate that revisions to  CSAT will be in place sometime in 2014. According to these officials,  improvements being considered for CSAT applications include a  reduction in the number of text-based responses and narrative  information required in the vulnerability assessment and the site security  plan; the inclusion of more drop-down menu options, which is also  expected to improve data analysis; and a reduction of the number of  repetitive questions, for example, in the site security plan. ISCD officials  noted that CSAT can be burdensome when facilities have to reenter data  from one document to the next and stated that they are looking to revise  CSAT so that information already entered on one document automatically  populates data fields covering the same question in the next. ISCD  officials also told us that one of the actions ISCD plans to take is to hold  three meetings, or roundtable discussions, with industry officials at  various locations beginning at the end of February 2013. They said that  these discussions are intended to obtain input from industry officials on  how CSAT can be improved. ISCD officials said that in making revisions  to the CSAT they will consider eliminating unnecessary data  requirements, but stated that they may decide to continue to request the  data—even if they are not used for risk tiering—because doing so may be  helpful to facilities as they prepare their security plans.

section_title: ISCD Does Not Seek Systematic Feedback on the Effectiveness of Its Outreach Efforts
paragraphs: ISCD seeks informal feedback on its outreach efforts but does not  systematically solicit feedback to assess the effectiveness of outreach  activities, and it does not have a mechanism to measure the  effectiveness of ISCD’s outreach activities. Trade association officials  reported that in general ISCD seeks informal feedback on its outreach  efforts and that members provide feedback to ISCD. Association officials  further reported that ISCD has encouraged association members to  contact local ISCD inspectors and has hosted roundtable discussions and  meetings where members of the regulated community provide feedback,  suggest improvements, or make proposals regarding aspects of the  CFATS program such as site security plans, alternative security  programs, gasoline storage site risks, and the personnel surety program.  Furthermore, according to ISCD officials, while feedback is solicited from  the regulated community generally on an informal basis, inspectors and  other staff involved in ISCD’s outreach activities are not required to solicit  feedback during meetings, presentations and assistance visits, and  inspectors are also not required to follow-up with the facilities after  assistance visits to obtain their views on the effectiveness of the  outreach.

ISCD, as part of its annual operating plan, has a priority for fiscal year  2013 to develop a strategic communications plan intended to address  both internal and external communication needs including industry  outreach. One goal in the plan is to maintain robust communication and  outreach. In addition, the annual operating plan contains 27 monitoring  and performance measures that address outreach program activities, but  only one of these calls for ISCD to solicit feedback to assess and  measure the effectiveness of the program. The NIPP states that when the  government is provided with an understanding of private sector  information needs, it can adjust its information collection, analysis,  synthesis, and dissemination activities accordingly. We have previously  reported on the benefits of soliciting systematic feedback. Specifically our  prior work on customer service efforts in the government, systematic  feedback from regulated facility owners and operators to among other  things, determine the kind and quality of services they want and also  determine their level of satisfaction with existing services including  outreach may benefit to organizations like ISCD that service the public.

As ISCD develops its strategic communication plan, options for gathering  feedback on outreach possibly could include using formal surveys  reviewed and approved by the Office of Management and Budget (OMB)   soliciting feedback from regulated under the Paperwork Reduction Act; owners and operators as a part of after-action reviews conducted at  assistance visits, meeting and presentations; working with trade  associations or other representatives of the regulated community to  design and conduct member surveys; or working with members of various  critical infrastructure sectors, such as the chemical or energy sectors, to  develop and conduct surveys of sector owners and operators. Given the  mixed perspectives of the trade associations we queried, systematic  feedback about outreach activities might better position ISCD to identify  problems and target changes to its outreach efforts, if necessary, or  improve CFATS program outcomes in general. Doing so would also be  consistent with Standards for Internal Control in the Federal Government,  which call for top level reviews of actual performance and the  establishment and review of performance measures and indicators.

section_title: Conclusions
paragraphs: ISCD has taken action to assign facilities to risk based tiers, revise its  process to review site security plans, and work with facilities to improve  security. However, three factors could affect program operations as ISCD  moves forward:

The first factor is a risk assessment approach that is not yet complete  because it does not consider all of the elements of risk called for by  the NIPP and the CFATS rule. ISCD has begun to take some actions  to develop a more robust risk assessment approach, but ISCD would  be better positioned to assess risk if it developed an overall plan, with  milestones and time frames, incorporating the results of the various  efforts to more fully address each of the three components of risk— consequence, threat, and vulnerability—and take actions to enhance  the current risk assessment approach. ISCD has commissioned an  expert panel to identify the strengths and weaknesses of the current  risk assessment approach and the results of the panel’s work could  help ISCD identify issues for further review and recommendations for  improvements. This effort and the results from it represent one  component of the various efforts ISCD will have to consider moving  forward to ensure that the risk assessment approach is complete per  the NIPP and the CFATS rule. After ISCD has developed and  completed its efforts to enhance its risk assessment approach by  using the results of the current expert panel’s efforts as well as  incorporating the issues we identified along with the Sandia National  Laboratories’ work on economic consequences, an independent peer  review would provide better assurance that ISCD can appropriately  identify and tier chemical facilities, better inform CFATS planning and  resource decisions; and provide the greatest return on investment  consistent with the NIPP and CFATS rule.

The second factor is ISCD’s ability to measure its progress reviewing  site security plans under its revised review process. ISCD has  developed measures to assess its progress reviewing site security  plans and has recently implemented a plan to measure various  aspects of the process. ISCD’s efforts appear to be a step in the right  direction, but, it will take time for ISCD to collect enough data to  develop baselines and begin measuring its performance. While it  could take years before ISCD can review and approve the site  security plans currently in its queue, ISCD intends to explore ways  that it can accelerate the process. Thus, we are not making  recommendations at this time.

The third factor is exploring opportunities to establish a mechanism to  systematically gather feedback to measure the effectiveness of ISCD  outreach efforts with facility owners and operators. This includes ISCD  efforts to communicate with owners and operators on various aspects  of the program, such as the development of site security plans, and  work with them to better understand how and whether data collection  requirements are burdensome and can be reduced. Doing so would  be consistent with (1) the NIPP which states that once the  government understands private sector information needs, it can  adjust its information collection, analysis, synthesis, and  dissemination activities accordingly, and (2) our past work on the  benefits of soliciting feedback to determine customer level of  satisfaction with existing services, including outreach. In addition, by  obtaining systematic feedback on its outreach efforts, ISCD might be  better positioned to (1) identify problems and target program changes,  if necessary, and generally improve ISCD efforts to communicate and  work with facility owners and operators, and (2) measure its  performance consistent with our Standards for Internal Control in the  Federal Government.

section_title: Recommendations for Executive Action
paragraphs: To better assess risk associated with facilities that use, process, or store  chemicals of interest consistent with the NIPP and the CFATS rule, we  recommend the Secretary of Homeland Security direct the Under  Secretary for NPPD, the Assistant Secretary for IP, and Director of ISCD  to take the following two actions: develop a plan, with timeframes and milestones, that incorporates the  results of the various efforts to fully address each of the components  of risk and take associated actions where appropriate to enhance  ISCD’s risk assessment approach consistent with the NIPP and the  CFATS rule, and conduct an independent peer review, after ISCD completes  enhancements to its risk assessment approach, that fully validates  and verifies ISCD’s risk assessment approach consistent with the  recommendations of the National Research Council of the National  Academies.

To enhance ISCD efforts to communicate and work with facilities, we  recommend that the Secretary of Homeland Security direct the Under  Secretary for NPPD, the Assistant Secretary for IP, and the Director of  ISCD to explore opportunities and take action to systematically solicit and  document feedback on facility outreach consistent with ISCD efforts to  develop a strategic communication plan.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Secretary of Homeland Security  for review and comment, which are reprinted in appendix II. In its written  comments, DHS agreed with our recommendations and stated that it has  efforts underway that will address them. Regarding our first  recommendation that ISCD develop a plan, with timeframes and  milestones, to fully address each of the components of risk and take  associated actions where appropriate to enhance ISCD’s risk assessment  approach consistent with the NIPP and the CFATS rule, DHS plans to   document all processes and procedures related to the risk assessment  approach and conduct an internal DHS review of the complete risk  assessment process, among others things, to ensure that all elements of  risk are included in the risk assessment approach. Regarding our second  recommendation that ISCD conduct an independent peer review, after it  completes enhancements to its risk assessment approach, that fully  validates and verifies ISCD’s risk assessment approach consistent with  the recommendations of the National Research Council of the National  Academies, DHS agreed that a peer review that includes validation and  verification steps would be a worthwhile endeavor, once any changes that  result from the aforementioned review of the risk assessment approach  are implemented. Regarding our third recommendation that ISCD explore  opportunities and take action to systematically solicit and document  feedback on facility outreach consistent with ISCD efforts to develop a  strategic communication plan, DHS plans to explore such opportunities to  make CFATS-related outreach efforts more effective for all stakeholders.  DHS also provided technical comments, which we incorporated as  appropriate.

We are sending copies of this report to interested congressional  committees and to the Secretary of the Department of Homeland  Security. In addition, the report will be available at no charge on GAO’s  website at http://www.gao.gov. If you or your staff members have any  questions about this report, please contact me at (202) 512-9610 or  caldwells@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made major contributions to this report are listed in  appendix III.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report is a follow-on engagement of work we completed in July 2012  about the actions the Department of Homeland Security’s (DHS)  Infrastructure Security Compliance Division (ISCD) took related to an  internal memorandum that cited an array of challenges—including human  capital and administrative issues—that hindered the implementation of  the Chemical Facility Anti-Terrorism Standards (CFATS) program. This  report discusses the extent to which DHS has assigned chemical facilities to risk-based tiers and assessed its  approach for doing so,  revised the process to review security plans, and communicated and worked with facilities to help improve security.

To determine the extent to which DHS has assigned chemical facilities to  risk-based tiers and assessed its approach for doing so, we reviewed  ISCD applications and documents including web-based Chemical  Security Assessment Tools (CSAT) applications—such as the Top- Screen and security vulnerability assessment—used to collect security  information from facilities, the ISCD risk assessment approach used to  determine a facility’s risk tier, policies and procedures on tiering, as well  as a sample copy of a facility’s Top Screen and security vulnerability  assessment. In addition, we outlined the risk tiering models associated  with different security issues—release, theft or diversion, and sabotage.  We also reviewed DHS memoranda detailing the challenges with the  release security issue model and ISCD’s work to improve it. We  compared our review to various criteria including the risk framework  outlined in the CFATS statute and rule, the National Infrastructure  Protection Plan (NIPP), and risk modeling best practices as outlined by  the National Research Council of the National Academy of Sciences to  determine if ISCD’s risk assessment approach comports with these  criteria and if not, where gaps or deficiencies exist. We also obtained data  from ISCD’s CSAT system that showed how many of the roughly 3,500  facilities, which ISCD has determined to be regulated by CFATS, were  placed in each of the 4 risk-based tiers and what their related security  issue(s) was. Because data in the system change regularly, these data  represent a snapshot of finally tiered facilities as of January 24, 2013. To  assess the reliability of the data we obtained from CSAT, we reviewed  system documentation, compared similar datasets for consistency, and  interviewed knowledgeable ISCD officials about system controls and  determined that the CSAT data were sufficiently reliable for the purposes  of this report. We also reviewed documents related to ISCD’s ongoing  review of the risk assessment approach including the statement of  objectives, task execution plan, and terms of reference and compared  these documents with the criteria for peer review as laid out by the  National Research Council of the National Academy of Sciences as well  as past GAO work on peer review.understanding of the risk assessment approach, we interviewed DHS and  contracting officials knowledgeable about the methodology behind the  models and their strengths and weaknesses. We also interviewed DHS  and contracting officials responsible for the review about the review’s  scope and progress, and to corroborate and confirm our understanding of  the review.

To corroborate and confirm our  To determine the extent to which DHS has revised its process to review  security plans, we reviewed and analyzed documents, where available,  including the CFATS statute and rule, the November 2011 internal  memorandum, DHS’s Risk-Based Performance Standards Guidance,  ISCD security plan review policies and procedures, security plan review  memoranda, and the instructions used by facilities to prepare and submit  security plans. As a part of our review of security plan documents, we  identified progress made and challenges encountered by ISCD as the  security plan review process evolved. To confirm our understanding of the  security plan review process and how it has evolved, we gathered and  analyzed statistics pertinent to the process to determine how many site  security plans had been reviewed, authorized, and approved as a percent  of the total number of security plans submitted. We then used the number  of plans approved, along with ISCD officials’ estimates of how many plans  they intend to approve per month beginning in calendar year 2013 to  estimate how long it might take ISCD to review and approve site security  plans for the approximately 3,500 currently regulated facilities. We also  interviewed ISCD officials about the security plan review process to  corroborate the information obtained from ISCD documents and the  results of our analyses.

To determine the extent to which DHS has communicated and worked  with facilities to help improve security, we reviewed documents (e.g.,  ISCD’s standard operating procedures for conducting compliance  assistance visits, inspections, and help desk support). We also reviewed  ISCD data on outreach activities—such as the type and number of field  visits to facilities and presentations to industry—for fiscal year 2007  through November 30, 2012 that we obtained from CHEMS. As noted  above, we determined that data from the CHEMS system were sufficiently  reliable for the purposes of this report. In addition, we contacted officials  representing 15 trade associations whose members are facilities  regulated by CFATS to obtain their perspectives on DHS’s efforts to  communicate and work with facility owners and operators to help improve  security. We selected these 15 trade associations because they are listed  in an annex to the NIPP as those with which DHS works on a regular  basis on chemical security matters. According to this annex, working with  these industry associations is a more manageable number of contact  points through which DHS can coordinate activities with a large number of  the asset owners and operators in the chemical sector. According to the  NIPP, a Sector Coordinating Council is the principal entity under which  owners and operators of critical infrastructure can coordinate with the  government on a wide range of protection activities and issues. The  Chemical Sector Coordinating Council represents a significant majority of  the owners and operators in the chemical sector. We sent a list of open- ended questions to representatives at each of the 15 trade associations  and received responses from 11 trade association representatives. We  analyzed, categorized, and summarized these responses by using a  systematic content analysis of the open-ended responses to determine  the trade association views. As a part of our analysis, two analysts  reviewed the responses, developed categories to be used for the content  analysis, and worked together to categorize each open-ended answer. A  third analyst reviewed this categorization and verified that the answers  were placed in the appropriate categories. Any disagreements regarding  the categorization of the answers were subsequently reconciled. The  information we obtained from the 11 trade associations that responded is  not generalizable to the universe of chemical facilities covered by CFATS;  however, it does provide insights into DHS efforts to perform outreach  and seek feedback on the implementation of the CFATS rule. We  compared the results of our analysis of the responses received from the  trade associations and other audit work related to ISCD’s outreach efforts  to various criteria, including CFATS statute and rule, the NIPP, past GAO  work on industry outreach, and internal control standards to determine if  DHS’s outreach efforts comport with these criteria, and if not, determined  where gaps exist. We also interviewed knowledgeable ISCD officials  regarding their outreach efforts.

We identified three limitations that should be considered when using our  results. First, as noted in our previous work, documentary evidence about  the development of the CFATS program is, for the most part, not  available. Program officials did not maintain records of key decisions  made in the early years of the program and the basis for these decisions.  This applies particularly to the risk tiering methodology and the security  plan review process. Furthermore, ISCD officials told us many of the  individuals involved in these decisions are no longer at ISCD. During  discussions, the current management team qualified that much of what  they told us about these decisions is their best guess of what happened  and why. This limits our ability to fully assess the risk model and compare  the original site security plan review process with the process currently in  place. Second, with regard to our work with facility owners and operators,  we limited our review to industry associations primarily due to time  constraints and the large number of owners and operators in the chemical  sector. As a result, we could not generalize our findings to the universe of  CFATS-regulated facility owners and operators. We mitigated this  limitation by contacting 15 industry associations that represented a wide  range of CFATS regulated chemical facilities. Third, given that ISCD had  only recently begun to approve security plans, our scope did not include  the facility compliance inspection process (which is based on the results  of the approved security plans).

We conducted this performance audit from October 2012 through April  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Comments from the Department of Homeland Security

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, John F. Mortin, Assistant  Director, and Ellen Wolfe, Analyst-in-Charge; Chuck Bausell; Jose  Cardenas; Michele Fejfar; Jeff Jensen; Tracey King; Marvin McGill; made  significant contributions to the work.

section_title: Related GAO Products
paragraphs: Critical Infrastructure Protection: An Implementation Strategy Could  Advance DHS’s Coordination of Resilience Efforts across Ports and Other  Infrastructure. GAO-13-11. Washington, D.C.: October 25, 2012.

Critical Infrastructure Protection: Summary of DHS Actions to Better  Manage Its Chemical Security Program. GAO-12-1044T. Washington,  D.C.: September 20, 2012.

Critical Infrastructure Protection: DHS Is Taking Action to Better Manage  Its Chemical Security Program, but It Is Too Early to Assess Results.  GAO-12-567T. Washington, D.C.: September 11, 2012.

Critical Infrastructure: DHS Needs to Refocus Its Efforts to Lead the  Government Facilities Sector. GAO-12-852. Washington, D.C.: August  13, 2012.

Critical Infrastructure Protection: DHS Is Taking Action to Better Manage  Its Chemical Security Program, but It Is Too Early to Assess Results.  GAO-12-515T. Washington, D.C.: July 26, 2012.

Critical Infrastructure Protection: DHS Could Better Manage Security  Surveys and Vulnerability Assessments. GAO-12-378. Washington, D.C.:  May 31, 2012.

Critical Infrastructure Protection: DHS Has Taken Action Designed to  Identify and Address Overlaps and Gaps in Critical Infrastructure Security  Activities. GAO-11-537R. Washington, D.C.: May 19, 2011.

Critical Infrastructure Protection: DHS Efforts to Assess and Promote  Resiliency Are Evolving but Program Management Could Be  Strengthened. GAO-10-772. Washington, D.C.: September 23, 2010.

Critical Infrastructure Protection: Update to National Infrastructure  Protection Plan Includes Increased Emphasis on Risk Management and  Resilience. GAO-10-296. Washington, D.C.: March 5, 2010.

The Department of Homeland Security’s (DHS) Critical Infrastructure  Protection Cost-Benefit Report. GAO-09-654R. Washington, D.C.: June  26, 2009.

Information Technology: Federal Laws, Regulations, and Mandatory  Standards to Securing Private Sector Information Technology Systems  and Data in Critical Infrastructure Sectors. GAO-08-1075R. Washington,  D.C.: September 16, 2008.

Risk Management: Strengthening the Use of Risk Management Principles  in Homeland Security. GAO-08-904T. Washington, D.C.: June 25, 2008.

Critical Infrastructure: Sector Plans Complete and Sector Councils  Evolving. GAO-07-1075T. Washington, D.C.: July 12, 2007.

Critical Infrastructure Protection: Sector Plans and Sector Councils  Continue to Evolve. GAO-07-706R. Washington, D.C.: July 10, 2007.

Critical Infrastructure: Challenges Remain in Protecting Key Sectors.  GAO-07-626T. Washington, D.C.: March 20, 2007.

Homeland Security: Progress Has Been Made to Address the  Vulnerabilities Exposed by 9/11, but Continued Federal Action Is Needed  to Further Mitigate Security Risks. GAO-07-375. Washington, D.C.:  January 24, 2007.

Critical Infrastructure Protection: Progress Coordinating Government and  Private Sector Efforts Varies by Sectors’ Characteristics. GAO-07-39.  Washington, D.C.: October 16, 2006.

Information Sharing: DHS Should Take Steps to Encourage More  Widespread Use of Its Program to Protect and Share Critical  Infrastructure Information. GAO-06-383. Washington, D.C.: April 17,  2006.

Risk Management: Further Refinements Needed to Assess Risks and  Prioritize Protective Measures at Ports and Other Critical Infrastructure.  GAO-06-91. Washington, D.C.: December 15, 2005.


Question-summary annotations: questions: question: How does DHS assign risk-based tiers?
answer: Since 2007, the Department of Homeland Security's (DHS) Infrastructure Security Compliance Division (ISCD) has assigned about 3,500 high-risk chemical facilities to risk-based tiers under its Chemical Facility Anti-Terrorism Standards (CFATS) program, but it has not fully assessed its approach for doing so.
child_questions: question: How is ISCD's assignment approach flawed?
answer: The approach ISCD used to assess risk and make decisions to place facilities in final tiers does not consider all of the elements of consequence, threat, and vulnerability associated with a terrorist attack involving certain chemicals.
child_questions: question: What factors does the risk assessment approach neglect?
answer: For example, the risk assessment approach is based primarily on consequences arising from human casualties, but does not consider economic consequences, as called for by the National Infrastructure Protection Plan (NIPP) and the CFATS regulation, nor does it consider vulnerability, consistent with the NIPP.

question: How will ISCD review its risk assessment approach?
answer: ISCD has begun to take some actions to examine how its risk assessment approach can be enhanced, including commissioning a panel of experts to assess the current approach, identify strengths and weaknesses, and recommend improvements. ISCD will need to incorporate the various results of these efforts to help them ensure that the revised risk assessment approach includes all elements of risk.
child_questions: question: How could an independent peer complement the ISCD's assessment approach?
answer: After ISCD has incorporated all elements of risk into its assessment approach, an independent peer review would provide better assurance that ISCD can appropriately identify and tier chemical facilities, better inform CFATS planning and resource decisions, and provide the greatest return on investment consistent with the NIPP.

questions: question: How has ISCD revised its security review process?
answer: DHS's ISCD has revised its process for reviewing facilities' site security plans--which are to be approved by ISCD before it performs compliance inspections--but it did not track data on the prior process to measure differences.
child_questions: question: Why was the process revised?
answer: The past process was considered by ISCD to be difficult to implement and caused bottlenecks in approving plans.
child_questions: question: How does the revised process compare to the old process?
answer: ISCD views its revised process to be a significant improvement because, among other things, teams of experts review parts of the plans simultaneously rather than sequentially, as occurred in the past.

question: What data will ISCD collect on its review process?
answer: Moving forward ISCD intends to measure the time it takes to complete reviews, but will not be able to do so until the process matures.
child_questions: question: How long will it take for ISCD to complete its reviews?
answer: GAO estimated that it could take another 7 to 9 years before ISCD is able to complete reviews on the approximately 3,120 plans in its queue which means that the CFATS regulatory regime, including compliance inspections, would likely be implemented in 8 to 10 years.

question: What is ISCD's current stance on its review process?
answer: ISCD officials said that they are exploring ways to expedite the process such as reprioritizing resources and streamlining inspection requirements.

questions: question: What actions has ISCD taken to work with the owners and operators?
answer: DHS's ISCD has also taken various actions to work with owners and operators, including increasing the number of visits to facilities to discuss enhancing security plans, but trade associations that responded to GAO's query had mixed views on the effectiveness of ISCD's outreach.

question: How does ISCD collect feedback on its outreach activities?
answer: ISCD solicits informal feedback from facility owners and operators on its efforts to communicate and work with them, but it does not have an approach for obtaining systematic feedback on its outreach activities.
child_questions: question: What opportunities does ISCD have to implement a systemic feedback collection process?
answer: ISCD's ongoing efforts to develop a strategic communication plan may provide opportunities to explore how ISCD can obtain systematic feedback on these activities.

question: What is the benefit of a systematic feedback collection approach?
answer: A systematic approach for gathering feedback and measuring the results of its outreach efforts could help ISCD focus greater attention on targeting potential problems and areas needing improvement.

questions: question: Why is the security of chemical facilities critical?
answer: Facilities that produce, store, or use hazardous chemicals could be of interest to terrorists intent on using toxic chemicals to inflict mass casualties in the United States.
child_questions: question: How does DHS regulate high-risk chemical facilities?
answer: As required by statute, DHS issued regulations that establish standards for the security of high-risk chemical facilities.

question: Why did DHS establish the CFATS program?
answer: DHS established the CFATS program to assess the risk posed by these facilities and inspect them to ensure compliance with DHS standards.
child_questions: question: What is ISCD's role in the CFATS program?
answer: ISCD, which manages the program, places high risk facilities in risk-based tiers and is to conduct inspections after it approves facility security plans.

questions: question: What did GAO review?
answer: GAO assessed the extent to which DHS has (1) assigned chemical facilities to tiers and assessed its approach for doing so, (2) revised its process to review facility security plans, and (3) communicated and worked with owners and operators to improve security.
child_questions: question: How did GAO collect information from DHS sources?
answer: GAO reviewed DHS reports and plans on risk assessments, security plan reviews, and facility outreach and interviewed DHS officials.

question: What data did GAO collect from chemical facilities?
answer: GAO also received input from 11 trade associations representing chemical facilities, about ISCD outreach.

question: To what extent is this review generalizable?
answer: The results of this input are not generalizable but provide insights.