Measuring Disaster Preparedness: FEMA Has Made Limited Progress in Assessing National Capabilities
Published: 2011-03-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: This testimony discusses the efforts of the Federal Emergency Management Agency (FEMA)--a component of the Department of Homeland Security (DHS)--to measure and assess national capabilities to respond to a major disaster. According to the Congressional Research Service, from fiscal years 2002 through 2010, Congress appropriated over $34 billion for homeland security preparedness grant programs to enhance the capabilities of state, territory, local, and tribal governments to prevent, protect against, respond to, and recover from terrorist attacks and other disasters. Congress enacted the Post-Katrina Emergency Management Reform Act of 2006 (Post-Katrina Act) to address shortcomings in the preparation for and response to Hurricane Katrina that, among other things, gave FEMA responsibility for leading the nation in developing a national preparedness system. The Post-Katrina Act requires that FEMA develop a national preparedness system and assess preparedness capabilities--capabilities needed to respond effectively to disasters--to determine the nation's preparedness capability levels and the resources needed to achieve desired levels of capability. Federal, state, and local resources provide capabilities for different levels of "incident effect" (i.e., the extent of damage caused by a natural or manmade disaster). FEMA's National Preparedness Directorate within its Protection and National Preparedness organization is responsible for developing and implementing a system for measuring and assessing national preparedness capabilities. The need to define measurable national preparedness capabilities is a well-established and recognized issue. For example, in December 2003, the Advisory Panel to Assess Domestic Response Capabilities noted that preparedness (for combating terrorism) requires measurable demonstrated capacity by communities, states, and private sector entities throughout the United States to respond to threats with well-planned, well-coordinated, and effective efforts. This is consistent with our April 2002 testimony on national preparedness, in which we identified the need for goals and performance indicators to guide the nation's preparedness efforts and help to objectively assess the results of federal investments. We reported that FEMA had not yet defined the outcomes of where the nation should be in terms of domestic preparedness. Thus, identifying measurable performance indicators could help FEMA (1) track progress toward established goals, (2) provide policy makers with the information they need to make rational resource allocations, and (3) provide program managers with the data needed to effect continual improvements, measure progress, and to enforce accountability. In September 2007, DHS issued the National Preparedness Guidelines that describe a national framework for capabilities-based preparedness as a systematic effort that includes sequential steps to first determine capability requirements and then assess current capability levels. According to the Guidelines, the results of this analysis provide a basis to identify, analyze, and choose options to address capability gaps and deficiencies, allocate funds, and assess and report the results. This proposed framework reflects critical practices we have identified for government performance and results. This statement is based on our prior work issued from July 2005 through October 2010 on DHS's and FEMA's efforts to develop and implement a national framework for assessing preparedness capabilities at the federal, state, and local levels, as well as DHS's and FEMA's efforts to develop and use metrics to define capability levels, identify capability gaps, and prioritize national preparedness investments to fill the most critical capability gaps. As requested, this testimony focuses on the extent to which DHS and FEMA have made progress in measuring national preparedness by assessing capabilities and addressing related challenges.
section_title: What GAO Found
paragraphs: In summary, DHS and FEMA have implemented a number of efforts with the goal of measuring preparedness by assessing capabilities and addressing related challenges, but success has been limited. DHS first developed plans to measure preparedness by assessing capabilities, but did not fully implement those plans. FEMA then issued the target capabilities list in September 2007 but has made limited progress in developing preparedness measures and addressing long-standing challenges in assessing capabilities, such as determining how to aggregate data from federal, state, local, and tribal governments. At the time of our review of FEMA's efforts in 2008 and in 2009, FEMA was in the process of refining the target capabilities to make them more measurable and to provide state and local jurisdictions with additional guidance on the levels of capability they need. We recommended in our April 2009 report that FEMA enhance its project management plan with, among other things, milestones to help it implement its capability assessment efforts; FEMA agreed with our recommendation. We reported in October 2010 that FEMA had enhanced its plan with milestones in response to our prior recommendation and that officials said they had an ongoing effort to develop measures for target capabilities--as planning guidance to assist in state and local assessments--rather than as requirements for measuring preparedness by assessing capabilities; FEMA officials had not yet determined how they plan to revise the list.

section_title: Letter
paragraphs: I appreciate the opportunity to participate in today’s hearing and to  discuss the efforts of the Federal Emergency Management Agency  (FEMA)—a component of the Department of Homeland Security (DHS)— to measure and assess national capabilities to respond to a major disaster.  According to the Congressional Research Service, from fiscal years 2002  through 2010, Congress appropriated over $34 billion for homeland  security preparedness grant programs to enhance the capabilities of state,  territory, local, and tribal governments to prevent, protect against, respond  to, and recover from terrorist attacks and other disasters. Congress  enacted the Post-Katrina Emergency Management Reform Act of 2006  (Post-Katrina Act) to address shortcomings in the preparation for and  response to Hurricane Katrina that, among other things, gave FEMA  responsibility for leading the nation in developing a national preparedness  system. The Post-Katrina Act requires that FEMA develop a national  preparedness system and assess preparedness capabilities—capabilities  needed to respond effectively to disasters—to determine the nation’s  preparedness capability levels and the resources needed to achieve  desired levels of capability. Figure 1 provides an illustration of how  federal, state, and local resources provide capabilities for different levels  of “incident effect” (i.e., the extent of damage caused by a natural or  manmade disaster). FEMA’s National Preparedness Directorate within its  Protection and National Preparedness organization is responsible for  developing and implementing a system for measuring and assessing  national preparedness capabilities.
The need to define measurable national preparedness capabilities is a  well-established and recognized issue. For example, in December 2003, the  Advisory Panel to Assess Domestic Response Capabilities noted that  preparedness (for combating terrorism) requires measurable  demonstrated capacity by communities, states, and private sector entities  throughout the United States to respond to threats with well-planned, well- coordinated, and effective efforts. This is consistent with our April 2002  testimony on national preparedness, in which we identified the need for  goals and performance indicators to guide the nation’s preparedness  efforts and help to objectively assess the results of federal investments.  We reported that FEMA had not yet defined the outcomes of where the  nation should be in terms of domestic preparedness. Thus, identifying  measurable performance indicators could help FEMA  track progress toward established goals,    provide policy makers with the information they need to make rational  resource allocations, and    provide program managers with the data needed to effect continual  improvements, measure progress, and to enforce accountability.
In September 2007, DHS issued the National Preparedness Guidelines that  describe a national framework for capabilities-based preparedness as a  systematic effort that includes sequential steps to first determine  capability requirements and then assess current capability levels.  According to the Guidelines, the results of this analysis provide a basis to  identify, analyze, and choose options to address capability gaps and  deficiencies, allocate funds, and assess and report the results. This  proposed framework reflects critical practices we have identified for  government performance and results. (See appendix 1 for an illustration  of the Guidelines and critical practices.)
My remarks today are based on our prior work issued from July 2005  through October 2010 on DHS’s and FEMA’s efforts to develop and  implement a national framework for assessing preparedness capabilities at  the federal, state, and local levels, as well as DHS’s and FEMA’s efforts to  develop and use metrics to define capability levels, identify capability  gaps, and prioritize national preparedness investments to fill the most  critical capability gaps. As requested, my testimony today focuses on the  extent to which DHS and FEMA have made progress in measuring national  preparedness by assessing capabilities and addressing related challenges.  To conduct our work, we analyzed documentation, such as FEMA’s  National Preparedness Guidelines and Target Capabilities List—a list of 37  capabilities that federal, state, and local stakeholders need to possess to  respond to natural or manmade disasters—and interviewed relevant DHS,  FEMA, state, and local officials. We conducted this work in accordance  with generally accepted government auditing standards. More detailed  information on our scope and methodology appears in our published  work.
In summary, DHS and FEMA have implemented a number of efforts with  the goal of measuring preparedness by assessing capabilities and  addressing related challenges, but success has been limited. DHS first  developed plans to measure preparedness by assessing capabilities, but  did not fully implement those plans. FEMA then issued the target  capabilities list in September 2007 but has made limited progress in  developing preparedness measures and addressing long-standing  challenges in assessing capabilities, such as determining how to aggregate  data from federal, state, local, and tribal governments. At the time of our  review of FEMA’s efforts in 2008 and in 2009, FEMA was in the process of  refining the target capabilities to make them more measurable and to  provide state and local jurisdictions with additional guidance on the levels  of capability they need. We recommended in our April 2009 report that  FEMA enhance its project management plan with, among other things,  milestones to help it implement its capability assessment efforts; FEMA  agreed with our recommendation. We reported in October 2010 that FEMA  had enhanced its plan with milestones in response to our prior  recommendation and that officials said they had an ongoing effort to  develop measures for target capabilities—as planning guidance to assist in  state and local assessments—rather than as requirements for measuring  preparedness by assessing capabilities; FEMA officials had not yet  determined how they plan to revise the list.
subsections: section_title: FEMA Has Made Limited Progress in Measuring Preparedness by Assessing Capabilities and Addressing Long- Standing Challenges
subsections: section_title: DHS Developed Plans for Assessing Capabilities, but Did Not Fully Implement Them
paragraphs: In July 2005, we reported that DHS had established a draft Target  Capabilities List that provides guidance on the specific capabilities and  levels of capability that FEMA would expect federal, state, local, and tribal  first responders to develop and maintain. We reported that DHS defined  these capabilities generically and expressed them in terms of desired  operational outcomes and essential characteristics, rather than dictating  specific, quantifiable responsibilities to the various jurisdictions. DHS  planned to organize classes of jurisdictions that share similar  characteristics—such as total population, population density, and critical  infrastructure—into tiers to account for reasonable differences in  capability levels among groups of jurisdictions and to appropriately  apportion responsibility for development and maintenance of capabilities  among levels of government and across these jurisdictional tiers.  According to DHS’s Assessment and Reporting Implementation Plan, DHS  intended to implement a capability assessment and reporting system based  on target capabilities that would allow first responders to assess their  preparedness to identify gaps, excesses, or deficiencies in their existing  capabilities or capabilities they will be expected to access through mutual  aid. In addition, this information could be used  to measure the readiness of federal civil response assets and the use of  federal assistance at the state and local level and  to provide a means of assessing how federal assistance programs are  supporting national preparedness.
In implementing this plan, DHS intended to collect preparedness data on  the capabilities of the federal government, states, local jurisdictions, and  the private sector to provide information about the baseline status of  national preparedness.
DHS’s efforts to implement these plans were interrupted by the 2005  hurricane season. In August 2005, Hurricane Katrina—the worst natural  disaster in our nation’s history—made final landfall in coastal Louisiana  and Mississippi, and its destructive force extended to the western Alabama  coast. Hurricane Katrina and the following Hurricanes Rita and Wilma— also among the most powerful hurricanes in the nation’s history— graphically illustrated the limitations at that time of the nation’s readiness  and ability to respond effectively to a catastrophic disaster, that is, a  disaster whose effects almost immediately overwhelm the response  capacities of affected state and local first responders and require outside  action and support from the federal government and other entities. In June  2006, DHS concluded that target capabilities and associated performance  measures should serve as the common reference system for preparedness  planning.
In September 2006, we reported that numerous reports and our work  suggest that the substantial resources and capabilities marshaled by  federal, state, and local governments and nongovernmental organizations  were insufficient to meet the immediate challenges posed by the  unprecedented degree of damage and the resulting number of hurricane  victims caused by Hurricanes Katrina and Rita. We also reported that  developing the capabilities needed for catastrophic disasters should be  part of an overall national preparedness effort that is designed to integrate  and define what needs to be done, where, based on what standards, how it  should be done, and how well it should be done. In October 2006,  Congress passed the Post-Katrina Act that required FEMA, in developing  guidelines to define target capabilities, ensure that such guidelines are  specific, flexible, and measurable. In addition, the Post-Katrina Act calls  for FEMA to ensure that each component of the national preparedness  system, which includes the target capabilities, is developed, revised, and  updated with clear and quantifiable performance metrics, measures, and  outcomes. We recommended, among other things, that DHS apply an all- hazards, risk management approach in deciding whether and how to invest  in specific capabilities for a catastrophic disaster; DHS concurred, and  FEMA said it planned to use the Target Capabilities List to assess  capabilities to address all hazards.
section_title: FEMA Issued the Target Capabilities List in September 2007 but Has Made Limited Progress in Developing Preparedness Measures and Addressing Long-standing Challenges in Assessing Capabilities
paragraphs: In September 2007, FEMA issued the Target Capabilities List to provide a  common perspective to conduct assessments to determine levels of  readiness to perform critical tasks and to identify and address any gaps or  deficiencies. According to FEMA, policymakers need regular reports on  the status of capabilities for which they have responsibility to help them  make better resource and investment decisions and to establish priorities.  Further, FEMA officials said that emergency managers and planners  require assessment information  to help them address deficiencies;  to identify alternative sources of capabilities (e.g., from mutual aid or  contracts with the private sector); and  to identify which capabilities should be tested through exercises.
Also, FEMA said that agencies or organizations that are expected to  supplement or provide capabilities during an incident need assessment  information to set priorities, make investment decisions, and position  capabilities or resources, if needed.
In April 2009, we reported that establishing quantifiable metrics for target  capabilities was a prerequisite to developing assessment data that can be  compared across all levels of government. At the time of our review,  FEMA was in the process of refining the target capabilities to make them  more measurable and to provide state and local jurisdictions with  additional guidance on the levels of capability they need. Specifically,  FEMA planned to develop quantifiable metrics—or performance  objectives—for each of the 37 target capabilities that are to outline  specific capability targets that jurisdictions (such as cities) of varying size  should strive to meet, being cognizant of the fact that there is not a “one  size fits all” approach to preparedness. However, FEMA has not yet  completed these quantifiable metrics for its 37 target capabilities, and it is  unclear when it plans to do so.
In October 2009, in responding to congressional questions regarding  FEMA’s plan and timeline for reviewing and revising the 37 target  capabilities, FEMA officials said they planned to conduct extensive  coordination through stakeholder workshops in all 10 FEMA regions and  with all federal agencies with lead and supporting responsibility for  emergency support-function activities associated with each of the 37  target capabilities. The workshops were intended to define the risk  factors, critical target outcomes, and resource elements for each  capability. The response stated that FEMA planned to create a Task Force  comprised of federal, state, local, and tribal stakeholders to examine all  aspects of preparedness grants, including benchmarking efforts such as  the Target Capabilities List. FEMA officials have described their goals for  updating the list to include establishing measurable target outcomes,  providing an objective means to justify investments and priorities, and  promoting mutual aid and resource sharing. In November 2009, FEMA  issued a Target Capabilities List Implementation Guide that described the  function of the list as a planning tool and not a set of standards or  requirements.
We reported in July 2005 that DHS had identified potential challenges in  gathering the information needed to assess capabilities, including  determining how to aggregate data from federal, state, local, and tribal  governments and others and integrating self-assessment and external  assessment approaches. In reviewing FEMA’s efforts to assess capabilities,  we further reported in April 2009 that FEMA faced methodological  challenges with regard to (1) differences in data available, (2) variations in  reporting structures across states, and (3) variations in the level of detail  within data sources requiring subjective interpretation. We recommended  that FEMA enhance its project management plan to include milestone  dates, among other things, a recommendation to which DHS concurred. In  October 2010, we reported that FEMA had enhanced its project  management plan.
Nonetheless, the challenges we reported in July 2005 and April 2009 faced  by DHS and FEMA, respectively, in their efforts to measure preparedness  and establish a system of metrics to assess national capabilities have  proved to be difficult for them to overcome. We reported that in October  2010, in general, FEMA officials said that evaluation efforts they used to  collect data on national preparedness capabilities were useful for their  respective purposes, but that the data collected were limited by data  reliability and measurement issues related to the lack of standardization in  the collection of data.
For example, FEMA’s Deputy Director for Preparedness testified in  October 2009 that the “Cost-to-Capabilities” (C2C) initiative developed by  FEMA’s Grant Programs Directorate (at that time already underway for 18  months) had a goal as a multiyear effort to manage homeland security  grant programs and prioritize capability-based investments. We reported  in October 2010, that as a result of FEMA’s difficulties in establishing  metrics to measure enhancements in preparedness capabilities, officials  discontinued the C2C program. Similarly, FEMA’s nationwide, multiyear  Gap Analysis Program implementation, proposed in March 2009, was “to  provide emergency management agencies at all levels of government with  greater situational awareness of response resources and capabilities.”  However, as we reported in October 2010, FEMA noted that states did not  always have the resources or ability to provide accurate capability  information into its Gap Analysis Program response models and  simulation; thus, FEMA had discontinued the program.
FEMA officials reported that one of its evaluation efforts, the State  Preparedness Report, has enabled FEMA to gather data on the progress,  capabilities, and accomplishments of a state’s, the District of Columbia’s,  or a territory’s preparedness program, but that these reports included self- reported data that may be subject to interpretation by the reporting  organizations in each state and not be readily comparable to other states’  data. The officials also stated that they have taken steps to address these  limitations by, for example, creating a Web-based survey tool to provide a  more standardized way of collecting state preparedness information that  will help FEMA officials validate the information by comparing it across  states.
We reported in October 2010 that FEMA officials said they had an ongoing  effort to develop measures for target capabilities—as planning guidance to  assist in state and local assessments —rather than as requirements for  measuring preparedness by assessing capabilities; FEMA officials had not  yet determined how they plan to revise the list and said they are awaiting  the completed revision of Homeland Security Presidential Directive 8,  which is to address national preparedness. As a result, FEMA has not yet  developed national preparedness capability requirements based on  established metrics to provide a framework for national preparedness  assessments. Until such a framework is in place, FEMA will not have a  basis to operationalize and implement its conceptual approach for  assessing federal, state, and local preparedness capabilities against  capability requirements to identify capability gaps for prioritizing  investments in national preparedness.
Mr. Chairman, this completes my prepared statement. I would be pleased  to respond to any questions that your or other Members of the Committee  may have at this time.
section_title: Contacts and Staff Acknowledgments
paragraphs: For further information about this statement, please contact William O.  Jenkins Jr., Director, Homeland Security and Justice Issues, at (202) 512- 8777 or jenkinswo@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. In addition to the contact named above, the following  individuals from GAO’s Homeland Security and Justice Team also made  major contributions to this testimony: Chris Keisling, Assistant Director;  John Vocino, Analyst-In-Charge; C. Patrick Washington, Analyst, and Lara  Miklozek, Communications Analyst.
section_title: Appendix I: National Preparedness Guidelines and Critical Practices for Performance Measurement
paragraphs: This appendix presents additional information on the Federal Emergency  Management Agency’s National Preparedness Guidelines as well as key  steps and critical practices for measuring performance and results.