Measuring Disaster Preparedness: FEMA Has Made Limited Progress in Assessing National Capabilities
Released: 2011-03-17T00:00:00

Letter

FEMA Has Made Limited Progress in Measuring Preparedness by Assessing Capabilities and Addressing Long- Standing Challenges

DHS Developed Plans for Assessing Capabilities, but Did Not Fully Implement Them

In July 2005, we reported that DHS had established a draft Target Capabilities List that provides guidance on the specific capabilities and levels of capability that FEMA would expect federal, state, local, and tribal first responders to develop and maintain. We reported that DHS defined these capabilities generically and expressed them in terms of desired operational outcomes and essential characteristics, rather than dictating specific, quantifiable responsibilities to the various jurisdictions. DHS planned to organize classes of jurisdictions that share similar characteristics—such as total population, population density, and critical infrastructure—into tiers to account for reasonable differences in capability levels among groups of jurisdictions and to appropriately apportion responsibility for development and maintenance of capabilities among levels of government and across these jurisdictional tiers. According to DHS’s Assessment and Reporting Implementation Plan, DHS intended to implement a capability assessment and reporting system based on target capabilities that would allow first responders to assess their preparedness to identify gaps, excesses, or deficiencies in their existing capabilities or capabilities they will be expected to access through mutual aid. In addition, this information could be used to measure the readiness of federal civil response assets and the use of federal assistance at the state and local level and to provide a means of assessing how federal assistance programs are supporting national preparedness.

In implementing this plan, DHS intended to collect preparedness data on the capabilities of the federal government, states, local jurisdictions, and the private sector to provide information about the baseline status of national preparedness.

DHS’s efforts to implement these plans were interrupted by the 2005 hurricane season. In August 2005, Hurricane Katrina—the worst natural disaster in our nation’s history—made final landfall in coastal Louisiana and Mississippi, and its destructive force extended to the western Alabama coast. Hurricane Katrina and the following Hurricanes Rita and Wilma— also among the most powerful hurricanes in the nation’s history— graphically illustrated the limitations at that time of the nation’s readiness and ability to respond effectively to a catastrophic disaster, that is, a disaster whose effects almost immediately overwhelm the response capacities of affected state and local first responders and require outside action and support from the federal government and other entities. In June 2006, DHS concluded that target capabilities and associated performance measures should serve as the common reference system for preparedness planning.

In September 2006, we reported that numerous reports and our work suggest that the substantial resources and capabilities marshaled by federal, state, and local governments and nongovernmental organizations were insufficient to meet the immediate challenges posed by the unprecedented degree of damage and the resulting number of hurricane victims caused by Hurricanes Katrina and Rita. We also reported that developing the capabilities needed for catastrophic disasters should be part of an overall national preparedness effort that is designed to integrate and define what needs to be done, where, based on what standards, how it should be done, and how well it should be done. In October 2006, Congress passed the Post-Katrina Act that required FEMA, in developing guidelines to define target capabilities, ensure that such guidelines are specific, flexible, and measurable. In addition, the Post-Katrina Act calls for FEMA to ensure that each component of the national preparedness system, which includes the target capabilities, is developed, revised, and updated with clear and quantifiable performance metrics, measures, and outcomes. We recommended, among other things, that DHS apply an all- hazards, risk management approach in deciding whether and how to invest in specific capabilities for a catastrophic disaster; DHS concurred, and FEMA said it planned to use the Target Capabilities List to assess capabilities to address all hazards.

FEMA Issued the Target Capabilities List in September 2007 but Has Made Limited Progress in Developing Preparedness Measures and Addressing Long-standing Challenges in Assessing Capabilities

In September 2007, FEMA issued the Target Capabilities List to provide a common perspective to conduct assessments to determine levels of readiness to perform critical tasks and to identify and address any gaps or deficiencies. According to FEMA, policymakers need regular reports on the status of capabilities for which they have responsibility to help them make better resource and investment decisions and to establish priorities. Further, FEMA officials said that emergency managers and planners require assessment information to help them address deficiencies; to identify alternative sources of capabilities (e.g., from mutual aid or contracts with the private sector); and to identify which capabilities should be tested through exercises.

Also, FEMA said that agencies or organizations that are expected to supplement or provide capabilities during an incident need assessment information to set priorities, make investment decisions, and position capabilities or resources, if needed.

In April 2009, we reported that establishing quantifiable metrics for target capabilities was a prerequisite to developing assessment data that can be compared across all levels of government. At the time of our review, FEMA was in the process of refining the target capabilities to make them more measurable and to provide state and local jurisdictions with additional guidance on the levels of capability they need. Specifically, FEMA planned to develop quantifiable metrics—or performance objectives—for each of the 37 target capabilities that are to outline specific capability targets that jurisdictions (such as cities) of varying size should strive to meet, being cognizant of the fact that there is not a “one size fits all” approach to preparedness. However, FEMA has not yet completed these quantifiable metrics for its 37 target capabilities, and it is unclear when it plans to do so.

In October 2009, in responding to congressional questions regarding FEMA’s plan and timeline for reviewing and revising the 37 target capabilities, FEMA officials said they planned to conduct extensive coordination through stakeholder workshops in all 10 FEMA regions and with all federal agencies with lead and supporting responsibility for emergency support-function activities associated with each of the 37 target capabilities. The workshops were intended to define the risk factors, critical target outcomes, and resource elements for each capability. The response stated that FEMA planned to create a Task Force comprised of federal, state, local, and tribal stakeholders to examine all aspects of preparedness grants, including benchmarking efforts such as the Target Capabilities List. FEMA officials have described their goals for updating the list to include establishing measurable target outcomes, providing an objective means to justify investments and priorities, and promoting mutual aid and resource sharing. In November 2009, FEMA issued a Target Capabilities List Implementation Guide that described the function of the list as a planning tool and not a set of standards or requirements.

We reported in July 2005 that DHS had identified potential challenges in gathering the information needed to assess capabilities, including determining how to aggregate data from federal, state, local, and tribal governments and others and integrating self-assessment and external assessment approaches. In reviewing FEMA’s efforts to assess capabilities, we further reported in April 2009 that FEMA faced methodological challenges with regard to (1) differences in data available, (2) variations in reporting structures across states, and (3) variations in the level of detail within data sources requiring subjective interpretation. We recommended that FEMA enhance its project management plan to include milestone dates, among other things, a recommendation to which DHS concurred. In October 2010, we reported that FEMA had enhanced its project management plan.

Nonetheless, the challenges we reported in July 2005 and April 2009 faced by DHS and FEMA, respectively, in their efforts to measure preparedness and establish a system of metrics to assess national capabilities have proved to be difficult for them to overcome. We reported that in October 2010, in general, FEMA officials said that evaluation efforts they used to collect data on national preparedness capabilities were useful for their respective purposes, but that the data collected were limited by data reliability and measurement issues related to the lack of standardization in the collection of data.

For example, FEMA’s Deputy Director for Preparedness testified in October 2009 that the “Cost-to-Capabilities” (C2C) initiative developed by FEMA’s Grant Programs Directorate (at that time already underway for 18 months) had a goal as a multiyear effort to manage homeland security grant programs and prioritize capability-based investments. We reported in October 2010, that as a result of FEMA’s difficulties in establishing metrics to measure enhancements in preparedness capabilities, officials discontinued the C2C program. Similarly, FEMA’s nationwide, multiyear Gap Analysis Program implementation, proposed in March 2009, was “to provide emergency management agencies at all levels of government with greater situational awareness of response resources and capabilities.” However, as we reported in October 2010, FEMA noted that states did not always have the resources or ability to provide accurate capability information into its Gap Analysis Program response models and simulation; thus, FEMA had discontinued the program.

FEMA officials reported that one of its evaluation efforts, the State Preparedness Report, has enabled FEMA to gather data on the progress, capabilities, and accomplishments of a state’s, the District of Columbia’s, or a territory’s preparedness program, but that these reports included self- reported data that may be subject to interpretation by the reporting organizations in each state and not be readily comparable to other states’ data. The officials also stated that they have taken steps to address these limitations by, for example, creating a Web-based survey tool to provide a more standardized way of collecting state preparedness information that will help FEMA officials validate the information by comparing it across states.

We reported in October 2010 that FEMA officials said they had an ongoing effort to develop measures for target capabilities—as planning guidance to assist in state and local assessments —rather than as requirements for measuring preparedness by assessing capabilities; FEMA officials had not yet determined how they plan to revise the list and said they are awaiting the completed revision of Homeland Security Presidential Directive 8, which is to address national preparedness. As a result, FEMA has not yet developed national preparedness capability requirements based on established metrics to provide a framework for national preparedness assessments. Until such a framework is in place, FEMA will not have a basis to operationalize and implement its conceptual approach for assessing federal, state, and local preparedness capabilities against capability requirements to identify capability gaps for prioritizing investments in national preparedness.

Mr. Chairman, this completes my prepared statement. I would be pleased to respond to any questions that your or other Members of the Committee may have at this time.

Contacts and Staff Acknowledgments

For further information about this statement, please contact William O. Jenkins Jr., Director, Homeland Security and Justice Issues, at (202) 512- 8777 or jenkinswo@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact named above, the following individuals from GAO’s Homeland Security and Justice Team also made major contributions to this testimony: Chris Keisling, Assistant Director; John Vocino, Analyst-In-Charge; C. Patrick Washington, Analyst, and Lara Miklozek, Communications Analyst.

Appendix I: National Preparedness Guidelines and Critical Practices for Performance Measurement

This appendix presents additional information on the Federal Emergency Management Agency’s National Preparedness Guidelines as well as key steps and critical practices for measuring performance and results.