National Preparedness: FEMA Has Made Progress, but Needs to Complete and Integrate Planning, Exercise, and Assessment Efforts
Released: 2009-05-29T00:00:00

Letter

Background

In 2005, Hurricane Katrina dramatically illustrated the adverse consequences that can occur when the nation is unprepared to respond effectively to a catastrophic disaster. Emergency preparedness strengthens the nation’s ability to prevent, protect, respond to, and recover from a natural disaster, terrorist attack, or other man-made disaster. It has received widespread attention and support from Congress, the President, and the Secretary of Homeland Security as manifested by legislation, presidential directives, the development of DHS policy documents, and grants to state and local governments. The lessons learned from the terrorist attacks of 9/11 and Hurricane Katrina focused attention on the need for preparedness programs that could (1) guide decisions on how to improve policies and plans that define roles and responsibilities across the broad spectrum of governmental and nongovernmental organizations involved in prevention, protection, response, and recovery activities and (2) help managers prioritize the use of finite resources to narrow gaps in needed capabilities.

FEMA—a component in DHS—is the federal agency responsible for leading the nation’s preparedness activities. In December 2003, the President issued guidance that called on the Secretary of Homeland Security to carry out and coordinate preparedness activities with public, private, and nonprofit organizations involved in such activities. In the wake of the problems that marked the response to Hurricane Katrina in 2005, Congress passed the Post-Katrina Act in October 2006. The act strengthened FEMA’s role within DHS and defined FEMA’s primary mission as: “to reduce the loss of life and property and protect the Nation from all hazards, including natural disasters, acts of terrorism, and other man-made disasters, by leading and supporting the nation in a risk-based, comprehensive emergency management system of preparedness, protection, response, recovery, and mitigation.” The act required FEMA to establish a national preparedness system for ensuring that the nation has the ability to deal with all hazards, including those incidents with catastrophic consequences. Among other things, the act directs FEMA to provide funding, training, exercises, technical assistance, planning, and other assistance to build tribal, local, state, regional, and national capabilities (including communications capabilities) necessary to respond to any type of disaster. It also requires FEMA to develop and coordinate the implementation of a risk-based all-hazards strategy for preparedness that builds those common capabilities necessary to any type of disaster, while also building the unique capabilities necessary to respond to specific types of incidents that pose the greatest risk to the United States. The act includes a number of other specific requirements including the development of quantifiable performance measurements to support each component of the national preparedness system, such as capabilities assessment, training, and exercises. The system provides a basis for improvements in policies, plans, and capabilities that aim to save lives and protect and preserve property.

National Preparedness Aims to Clarify Roles and Responsibilities and Close Gaps in Disaster Response Capability

DHS has defined national preparedness as a continuous cycle that involves four main elements: (1) policy and doctrine, (2) planning and resource allocation, (3) training and exercises, and (4) an assessment of capabilities and reporting. The following is a brief description of each element of the system.

Policy: This element involves ongoing management and maintenance of national policy and doctrine for operations and preparedness. Disaster response is primarily handled by local or tribal governments with the state and federal governments and private and nonprofit sectors playing supporting and ad hoc roles, respectively, as needed or as requested. One of the lessons learned from Hurricane Katrina is that a lack of clarity regarding roles and responsibilities across these levels of government and sectors can result in a less coordinated national response and delay the nation’s ability to provide life-saving support when needed. Broadly speaking, FEMA’s role, in cooperation with other federal and nonfederal entities, is to define the roles and responsibilities for all response stakeholders so that each understands how it supports the broader national response. This approach calls for a national response based on partnerships at and across all levels.

Planning and resource allocation: This element involves application of common planning processes and tools by government officials, working with the private sector, nongovernmental organizations, and individual citizens to identify requirements, allocate resources, and build and maintain coordinated capabilities that are prioritized based upon risk. Among other things, this element involves developing planning processes so that roles and responsibilities of stakeholders are clearly defined for specific homeland security scenarios, such as a hurricane or a terrorist attack involving nuclear or radiological weapons.

Training and exercises: Exercises provide opportunities to test plans and improve proficiency in a risk-free environment. Exercises assess the adequacy of capabilities as well as the clarity of established roles and responsibilities. Short of performance in actual operations, exercise activities provide the best means to evaluate returns on homeland security investments. The Post-Katrina Act requires FEMA to carry out a National Training Program and National Exercise Program. On January 26, 2007, the National Security Council and the HSC approved the establishment of a new iteration of the National Exercise Program to conduct exercises to help senior federal government officials prepare for catastrophic crises ranging from terrorism to natural disasters. Well-designed and executed exercises can improve interagency coordination and communications, highlight capability gaps, and identify opportunities for improvement. Tracking corrective actions resulting from exercises is a key step in the process.

Assessing capabilities and reporting: According to the Post-Katrina Act, FEMA is required to develop a comprehensive assessment system to assess the nation’s prevention capabilities and overall preparedness. A key part of the system involves the development of quantifiable standards and metrics—called target capabilties—that can be used to assess existing capability levels compared with target capability levels. The act requires FEMA to include the results of its comprehensive assessments in an annual Federal Preparedness Report to Congress. To assist in this effort, FEMA is to receive annual State Preparedness Reports from all 50 states, the District of Columbia, and 5 territories that receive DHS preparedness assistance.

FEMA’s National Preparedness Directorate has primary responsibility for carrying out the key elements of the national preparedness system, in coordination with other federal, state, local, tribal, nonprofit, and private- sector organizations. The directorate includes the National Integration Center and the Office of Preparedness Policy, Planning, and Analysis (PPPA). The National Preparedness Directorate and FEMA’s Disaster Operations Directorate share responsibility for ensuring plans that describe roles and responsibilities are developed. FEMA’s National Exercise Division—a division of the National Integration Center—leads exercise activities. Finally, PPPA is responsible for assessing capabilities. See figure 2 for an organizational chart of select FEMA components involved in preparedness programs.

Developing and Integrating the Elements of the National Preparedness System Is a Challenging Responsibility

FEMA faces two main challenges in developing and integrating the elements of the national preparedness system.

For example, see GAO, National Response Framework: FEMA Needs Policies and Procedures to Better Integrate Non-Federal Stakeholders in the Revision Process, GAO-08-768 (Washington, D.C.: June 11, 2008) and Voluntary Organizations: FEMA Should More Fully Assess Organization’s Mass Care Capabilities and Update the Red Cross Role in Catastrophic Events, GAO-08-823 (Washington, D.C.: Sept. 18, 2008).

See Homeland Security Advisory Council, Top Ten Challenges Facing The Next Secretary of Homeland Security (Washington, D.C.: Sept. 11, 2008). Among other things, this report determined that the work of strengthening disaster response capabilities is incomplete, in part, because DHS will need to ensure involvement of homeland security partners in building a bottom-up approach of organization and response as it establishes national planning efforts.

The National Council on Disability is an independent federal agency and is composed of 15 members appointed by the President. It provides advice to the President, Congress, and executive branch agencies to promote policies, programs, practices, and procedures that guarantee equal opportunity for all individuals with disabilities, regardless of the nature or severity of the disability and to empower individuals with disabilities to achieve economic self-sufficiency, independent living, and inclusion and integration into all aspects of society. involves these preparedness stakeholders in plans that define roles a responsibilities, exercises, and assessments of capabilities.

6 U.S.C. § 315.

March 2008 half of the staff positions in the PPPA office were vacant and these positions were not filled until November 2008, according to FEMA officials. In addition, FEMA did not hire a permanent director for the PPPA office until October 2008. Reorganization of the National Exercise Program also brought about staffing changes, according to FEMA officials. Among other things, in the fall of 2008 FEMA hired a new director for the National Exercise Division—a position that had been vacant for several months, according to FEMA officials. Fig. 3 illustrates the organizational changes related to the National Exercise Program.

Most Policies That Define Roles and Responsibilities and Planning Processes Have Been Completed, but Work Remains to Complete Emergency Plans

Defining roles and responsibilities is a key step in developing the national preparedness system. While most key policies that define roles and responsibilities have been completed, 68 percent (49 of 72) of plans that operationalize such policies have not been completed. Lessons learned from Hurricane Katrina and emergency response exercises demonstrated the need for the development of complete policies and plans to address potential problems with stakeholders not understanding their roles and responsibilities in response to a catastrophic disaster. Although best practices for program management state that a program management plan is an essential tool for implementing a program, FEMA, in coordination with DHS and other federal entities, has not yet fully developed such a plan to help ensure the development and integration of policies and plans that define roles and responsibilities and planning processes for emergency response.

Defining Roles and Responsibilities Is Key to Developing the Preparedness System

Legislation and presidential directives call for the development of policies and plans that define roles and responsibilities, which is key to FEMA’s ability to develop the preparedness system. For example, the Homeland Security Act of 2002, as amended by the Post-Katrina Act, requires FEMA to consolidate existing federal government emergency response plans into a single, coordinated National Response Plan, now known as the National Response Framework. To develop the preparedness system, FEMA is to partner and coordinate with key stakeholders, including other DHS components as well as other federal, state, and local entities. For example, the Post-Katrina Act requires the FEMA Administrator, under the leadership of the Secretary of Homeland Security, to coordinate with other agencies and offices in DHS to take full advantage of their range of available resources. In addition, the Post-Katrina Act requires FEMA to coordinate with other federal departments and agencies and the National Advisory Council to develop and refine key national policy documents that broadly define roles and responsibilities, including the National Incident Management System (NIMS) and the NRF.

Plans developed using established planning processes operationalize policy documents, such as the NRF, by providing additional details on the roles and responsibilities for each individual and organization that may be involved in responding to high-risk or catastrophic incidents. For example, during the planning process, FEMA may need to coordinate with other federal departments and agencies—such as the Department of Defense (DOD), Federal Bureau of Investigation, and other components in DHS—to define roles and responsibilities for responding to a chemical attack. Per the Post-Katrina Act, each federal agency with responsibilities under the NRF is responsible for developing its own operational plans.

The act further requires the President to certify to selected committees of Congress on an annual basis that each federal agency has complied with statutory requirements in the development of its operational plan, including a requirement that the plan “be coordinated under a unified system with a common terminology, approach, and framework.” Although the Post-Katrina Act does not charge FEMA with developing or certifying the federal agency plans, FEMA is statutorily responsible for the basic architecture of the national preparedness system, in coordination with other federal departments and agencies, among others. This principle of integration and coordination is also embodied in Homeland Security Presidential Directive 8 Annex 1 (HSPD 8 Annex 1), which tasked DHS with developing a national planning system to develop and integrate plans that define preparedness roles and responsibilities, both horizontally across federal departments and agencies (i.e., integration of plans that have been developed by more than one federal department or agency) and vertically with state and local emergency response plans (i.e., integration of plans that have been developed by more than one level of government). Thus, the federal preparedness framework depends on DHS’s—and, in particular, FEMA’s—ability to coordinate with its federal department and agency partners to ensure that their plans are integrated in a way that avoids duplication of effort and confusion during an interagency response.

Most Key Policies That Define Roles and Responsibilities and Planning Processes Have Been Completed

DHS, FEMA, and other federal entities with a role in national preparedness have completed most of the key policies that broadly define roles and responsibilities and planning processes for developing more detailed emergency plans. Among the 50 policies that define roles and responsibilities or planning processes, 42 have been completed, 2 have been partially completed, and the remaining 6 are incomplete.

A more detailed breakdown of the 50 policies shows that 46 define roles and responsibilities and 4 define planning processes for developing emergency plans. Forty of the 46 policies that define roles and responsibilities have been completed. Among the policies that have been completed, for example, DHS issued the revised NIMS in December 2008 to further clarify roles and responsibilities when multiagency, intergovernmental entities are involved in a response and to address, in part, the confusion about roles and responsibilities that resulted in a poor response to Hurricane Katrina. Key components of the NRF have also been completed, including the base NRF document, 15 Emergency Support Function Annexes, and 8 Support Annexes. However, other components of the NRF have not been completed, such as the 4 Partner Guides that are to provide abbreviated descriptions of the key roles and responsibilities of specific federal, state, local, and private sector and nongovernmental stakeholders under the NRF. Also uncompleted is the development of new Joint Field Office (JFO) guidance under the NRF which, according to FEMA officials, is to provide functional guidance for the organization and staffing of JFOs, as well as their establishment, operation, and demobilization. In addition, DHS has not completed the National Homeland Security Plan, which is to serve as an overarching strategic plan to guide national efforts to execute the National Strategy for Homeland Security. For details on, and the status of, all 46 policies that are to define roles and responsibilities, see table 4 in appendix II.

In addition to the 46 policies that define roles and responsibilities, 4 other policies are to define planning processes for developing emergency plans. Two of these 4 policies have been completed. According to officials, FEMA issued planning guidance for its operational planners in March 2009 and will update this guidance each fiscal year. The other 2 policies that define planning processes have been partially completed—1 has been issued as an interim policy (Comprehensive Preparedness Guide 301) while the other has been drafted (Integrated Planning System) and is being used by federal interagency incident management planners although it has not been publicly released. Additional details describing these 4 policies and their status are included in table 5 in appendix II.

Although Action Has Been Taken to Develop and Complete Some Plans That Define Roles and Responsibilities, 68 Percent of the Plans Have Not Been Completed

While DHS, FEMA, and other federal entities with a role in national preparedness have taken action to develop and complete some plans that detail and operationalize roles and responsibilities for federal and nonfederal entities, these entities have not completed 68 percent of the plans required by existing legislation, presidential directives, and policy documents as of April 2009. Specifically, of the 72 plans we identified, 20 have been completed (28 percent), 3 have been partially completed (that is, an interim or draft plan has been produced—4 percent), and 49 (68 percent) have not been completed. Detailed plans supplement and operationalize key policy documents. Among the plans that have been completed, FEMA published the Pre-Scripted Mission Assignment Catalog in 2008, which defines roles and responsibilities for 236 mission assignment activities to be performed by federal government entities, at the direction of FEMA, to aid state and local jurisdictions during a response to a major disaster or an emergency. See table 6 in appendix II for additional details on the other 19 plans that have been completed.

One of the three plans that FEMA has partially completed is the Federal Contingency Plan—New Madrid Seismic Zone Catastrophic Earthquake. This plan addresses major issues the federal government expects to encounter if a catastrophic earthquake occurs in the New Madrid Seismic Zone with no warning. FEMA published this plan in an interim form in June 2008 and intends to finalize it by May 2010. While FEMA has engaged in significant planning efforts regarding threats that are specific to certain regions, such as hurricanes and earthquakes, through its Catastrophic Disaster Planning Initiative, those planning efforts are ongoing and have not been concluded. See table 6 in appendix II for additional details on the other two plans that have been partially completed.

Among the 49 plans that have not been completed are the NRF incident annexes for terrorism and cyber incidents as well as the NRF incident annex supplements for catastrophic disasters and mass evacuations. The NRF incident annexes and incident annex supplements are to address the roles and responsibilities and unique aspects of how the United States responds to broad incident types. In addition, operational plans for responding to the consolidated national planning scenarios, as called for in HSPD 8 Annex 1, remain outstanding. For additional details and the status of each of these plans, as well as other plans that are to define roles and responsibilities, see table 6 in appendix II.

Developing plans to operationalize policies that define roles and responsibilities is one key to an effective response. According to DHS, effective response hinges upon well-trained leaders and responders who have invested in response preparedness, developed engaged partnerships, and are able to achieve shared objectives. Until outstanding policies and plans, especially those that are new, are completed, FEMA, in coordination with DHS and other federal departments and agencies, cannot provide associated training on such policies and plans, and, relatedly, cannot validate the new policies, plans, and training though exercises—the next step in the national preparedness system cycle.

According to FEMA, certain plans that have yet to be completed are refinements of existing plans, and training has been provided on the existing plan. However, such existing training will need to be adapted and modified to enable response stakeholders to be trained on the revised plans. Incomplete policies and plans, especially those that are new, and the resulting lack of associated training and validation through exercises, increase the risk that response to an incident may be disjointed, delayed, or ineffective because stakeholders may not understand their roles and responsibilities.

The issue of completing emergency policies and plans that define roles and responsibilities is not new. Prior to Hurricane Katrina, based on a summary of lessons learned from exercises conducted in fiscal year 2005, DHS determined that plans for specific incidents were potentially needed to clarify how the National Response Plan (now the NRF) would be implemented under several types of domestic scenarios, such as the threat of an improvised nuclear device detonation, large-scale biological events, and suicide bombings. In February 2006, the White House issued its report on lessons learned from the response to Hurricane Katrina and concluded that two of four critical deficiencies in the response involved policies and plans, or the lack thereof, that were to detail roles and responsibilities. Among other things, the report noted that federal departments and agencies were required to develop supporting operational plans and standard operating procedures for national response activities, but in almost all cases, these required plans and procedures were either nonexistent or still under development. In addition, the report stated that additional structural deficiencies in the national preparedness system included weak regional response planning and coordination structures. Further, in our September 2006 report on Hurricane Katrina, we recommended that FEMA develop detailed and robust operational implementation plans for the National Response Plan (now NRF) and its Catastrophic Incident Annex and Supplement in preparation for and response to future catastrophic disasters.

In addition, in October 2006, the Post-Katrina Act required FEMA to develop prescripted mission assignments and each federal agency with NRF responsibilities to develop operational plans and corresponding capabilities in support of the NRF to ensure a coordinated federal response. In December 2007, the President signed HSPD 8 Annex 1, which called for the development of an integrated set of scenario-based response plans, including federal agency operational plans that are also needed to satisfy the Post-Katrina Act. More recently, in September 2008, the Homeland Security Advisory Council (HSAC) for DHS identified the work of strengthening the nation’s disaster response capabilities, which includes developing and publishing outstanding emergency response policies and plans that define roles and responsibilities for national preparedness, as 1 of the top 10 DHS challenges for the incoming presidential administration.

Despite Exercises Demonstrating the Need for the Development of Complete Policies and Plans, FEMA Has Not Fully Developed a Program Management Plan for Their Development and Integration

Recent Exercises Have Demonstrated the Importance of the Development of Complete Policies and Plans to Define Roles and Responsibilities

In addition to the lessons learned from Hurricane Katrina, recent exercises have demonstrated the ongoing need for the development of complete policies and plans that define roles and responsibilities for national preparedness. For example, we reviewed 16 after-action reports for national or principal level or equivalent exercises that were conducted from 2005 to 2008, and all of the reports (16 of 16) called for further clarification of roles and responsibilities across federal departments and agencies or between federal and nonfederal organizations, as shown in the following examples.

As a result of a February 2007 exercise that tested a joint response by federal agencies to an attack using improvised explosive devices, HSC determined that the federal government needed to further refine the delineation of roles and responsibilities for the Departments of Defense, Justice, and Homeland Security. The same exercise also identified the need for the federal government to increase outreach to governors in order to achieve a common understanding of federal and state governments’ respective roles and responsibilities during such an incident. Finally, this exercise demonstrated that policies and plans were needed to clearly articulate, prior to an incident occurring, the circumstances in which the use of military assets and resources for civilian response is appropriate. Not addressing these recommended corrective actions, among others, may result in less efficient and effective responses.

In a September 2007 exercise, the HSC identified the need for senior federal officials to be rapidly informed of the possible courses of action available to them in responding to a major disaster or an emergency. To do so, HSC asserted that the senior federal officials’ respective organizations should have in place detailed plans to inform and execute the senior federal officials’ decisions, as such plans would serve to better integrate interagency response activities. HSC made this recommendation in September 2007, and it was acted on by the President through the establishment of HSPD 8 Annex 1 in December 2007, more than a year after the Post-Katrina Act called for the development of federal agency operational plans.

FEMA Has Not Established a Program Management Plan for the Development and Integration of Outstanding Emergency Response Policies and Plans

FEMA has not established a program management plan, in coordination with DHS and other federal entities, to ensure the development and integration of outstanding policies and plans that are to define roles and responsibilities and planning processes. The Post-Katrina Act makes the FEMA Administrator, in coordination with other entities, responsible for the development of the national preparedness system. According to the National Preparedness Guidelines, the national preparedness system, in part, consists of the policies and plans that define roles and responsibilities and planning processes for developing emergency plans. Although the Post-Katrina Act requires federal agencies to develop their own operational plans, those plans are to be “coordinated under a unified system with a common terminology, approach, and framework.” This coordination and unification is central to FEMA’s mission as the lead agency in charge of national preparedness, and it requires that the policies and plans that have been called for are developed and integrated so that emergency response roles and responsibilities and planning processes are fully defined and implemented.

Best practices for program management, established by the Project Management Institute in The Standard for Program Management, state that managing a program includes, among other things, (1) establishing clear and achievable objectives; (2) balancing the competing demands for quality, scope, time, and cost; and, (3) adapting the specifications, plans, and approach to the different concerns and expectations of the various stakeholders involved in the program’s projects. A key step in managing a program involves developing a program management plan, which is an approved document that defines how a program will be executed, monitored, and controlled. The program management plan defines the tactical means by which the program will be carried out. According to The Standard for Program Management, a program management plan should, among other things: identify the specific schedule of activities that need to be performed to complete and identify dependencies among policy and planning development activities; identify the types and quantities of resources required to perform, and amount of time needed to complete, all policy and planning development activities; analyze activity sequences, durations, resource requirements, and schedule constraints to create and update the policy and planning project schedules; and control for changes to the project schedules precipitated by outside forces.

Because FEMA has not established, in coordination with DHS and other federal entities, such a plan to ensure the development and integration of outstanding policies and plans, it is unclear when the full set of policies and plans will be completed, and FEMA cannot determine whether it or other entities with policy and plan development responsibilities, such as DHS, are on schedule. Further, because FEMA cannot determine whether other entities with policy and plan development responsibilities are on schedule, it cannot determine when and how it will integrate into the national preparedness cycle the range of policies and plans required by legislation, presidential directives, and other policy. Based in part on our work, in February 2009 FEMA officials acknowledged that a program management plan should be established. Without active utilization of a program management plan, FEMA, in coordination with DHS and other federal entities, may experience unforeseen delays in completing its efforts to develop and integrate these policies and plans that define roles and responsibilities and planning processes. While FEMA, in coordination with DHS and other federal entities, may experience unanticipated or uncontrollable delays in developing outstanding policies and plans, it would be better positioned to identify the effect of those delays and assess measures to mitigate them with a program management plan in place and utilized.

FEMA Has Taken Actions to Implement the National Exercise Program, but Faces Challenges in Meeting Statutory and Program Requirements

FEMA has developed guidance to implement the National Exercise Program; however, it faces challenges in meeting statutory and program requirements in conducting the program. These challenges have arisen because FEMA lacks procedures that detail how it will work with federal entities and monitor states to ensure these entities carry out program requirements. In addition, FEMA faces challenges in measuring the effectiveness of the program because the databases it uses to measure program performance are incomplete.

FEMA Has Developed Guidance and Other Tools to Assist in Designing and Implementing the National Exercise Program

Exercises are a key element of the national preparedness system. The purpose of the National Exercise Program is to test and improve the nation’s ability to prevent, prepare for, and respond to events such as terrorist attacks and natural and man-made disasters. To meet this purpose, exercises should test existing capabilities against desired, or target capabilities as well as verify and validate policies and plans that define roles and responsibilities.

Developing and implementing the National Exercise Program is a difficult task because the magnitude of the effort involves coordinating with and relying on the cooperation of other DHS components such as the Coast Guard, numerous federal entities such as the Homeland Security Council (HSC)—which is responsible for coordinating federal interagency homeland security policy—and state governments, among others. This coordination is especially critical at the federal level because FEMA lacks the authority to compel federal agencies to comply with program requirements. At the state level, FEMA is able to use its grant programs to ensure that states follow program guidelines. Since 2007, FEMA has taken a number of actions to implement the National Exercise Program. Several of these actions are summarized in table 1.

FEMA has also identified four tiers of exercises that comprise the National Exercise Program (see fig. 4). Among the exercises that involve federal interagency coordination are Tier I National Level Exercises, which are operations-based exercises that evaluate existing national plans and policies, in concert with federal and nonfederal entities; and Principal Level Exercises, which are discussion-based exercises among senior federal officials that examine emerging issues.

Officials at all six states we visited cited actions taken by FEMA to implement the National Exercise Program as positive contributions to their exercise efforts. For example, officials in five of the six states indicated that HSEEP guidance was beneficial because it establishes consistency in how exercises are designed and conducted. State officials who were involved in a National Level Exercise indicated that it not only allowed them to test and validate local emergency response plans, but also gave them the opportunity to meet federal stakeholders as well as first responders from neighboring counties and cities, thus enabling them to establish working relationships before an event occurs. Similarly, exercise planners in New York noted that a benefit of FEMA’s regional training and exercise plan workshop was the positive relations that were developed with FEMA officials.

States Have Also Taken Actions to Implement Exercise Programs

In addition to actions by FEMA, the six states we visited have also taken actions to implement exercise programs (FEMA considers exercises conducted by states to be Tier IV exercises.) For example, exercise planners in Washington conducted a half-day senior leadership workshop for federal, state, and local officials in May 2008 that identified a number of issues related, among other things, to intergovernmental communications during an incident. Based on the workshop, officials in Washington identified corrective actions that could be taken. For example, they identified the need to increase postdisaster communications to help speed recovery efforts and said they were taking corrective actions to address this issue. In California, state exercise planners involved planners from other departments or other states in their design and planning workshops as a way to share information on exercise design and conduct, thereby increasing the interagency participation in their exercise program efforts. According to the workshop sponsors, participants benefited from the multidisciplinary participation in the conference. In Illinois, the state used real-world events, such as sporting events, to test or “exercise” response by local law enforcement and first responders. State exercise officials said these efforts were an effective supplement to their regular exercise program.

FEMA Faces Challenges in Meeting Statutory and Program Requirements in Conducting and Measuring the Effectiveness of the National Exercise Program

While FEMA has taken actions to implement the National Exercise Program, it faces challenges in meeting statutory and program requirements in conducting and measuring the effectiveness of the program. First, FEMA does not have procedures in place to detail how it is to (1) work with other federal entities and (2) monitor states to help ensure that these entities promptly prepare after-action reports and track and implement corrective actions for federal- and state-level exercises. Second, the National Exercise Program’s ability to simulate a catastrophic event to strain, or “stress”, the preparedness system is limited. Third, FEMA lacks data to measure the effectiveness and progress of the National Exercise Program.

FEMA Does Not Have Procedures to Help Ensure that After-Action Reports from National and Principal Level Exercises Are Promptly Prepared and That Corrective Actions Are Tracked and Implemented

FEMA must design the National Exercise Program to provide, among other things, for the prompt development of after-action reports and plans for quickly incorporating lessons learned into future operations. In addition, the Post-Katrina Act requires FEMA to establish a program to conduct remedial action tracking and long-term trend analyses. According to the implementation plan for the National Exercise Program, after-action reports for Tier I exercises (National Level Exercises or Principal Level Exercises) must be issued within 180 days, or 6 months, after the completion of an exercise, and the release of an after-action report should not be delayed to reach consensus on all issues identified during the exercise. FEMA executes Tier I exercises in coordination with HSC and others. Although the Post-Katrina Act does not give FEMA the authority to compel other federal entities to comply with the objectives of the National Exercise Program, it places responsibility for implementing the National Exercise Program on FEMA, in coordination with other appropriate federal agencies and other entities. Therefore, it is incumbent on FEMA to coordinate with HSC and other federal entities to better ensure that FEMA obtains the information it needs to meet its statutory responsibility to track corrective actions. However, FEMA has not ensured that after-action reports for Tier 1 exercises are issued in a prompt manner, nor has it tracked and documented implementation of corrective actions for such exercises. These challenges occurred, in part, because FEMA has not established procedures that detail how the agency will work with other federal entities to ensure National Exercise Program requirements are met.

For example, FEMA conducted a Tier I National Level Exercise—Top Officials 4 (TOPOFF 4)—in October 2007, but as of February 2009, or more than 15 months later, FEMA had not yet issued the after-action report or tracked and implemented corrective actions. When an after-action report on an exercise is delayed and not provided to stakeholders, the “lessons learned” from the exercise diminish in importance, limiting stakeholders’ ability to make improvements in preparedness. In February 2009, FEMA officials stated that the draft after-action report for TOPOFF 4 had been written, and various departments and agencies have approved the report, but it was not approved by DHS prior to the change in administration related to the 56th Presidential Inauguration. According to FEMA officials, a complicating factor in releasing the report is the political sensitivity of information, as those who write after-action reports may face internal pressure not to identify weaknesses in an entity’s emergency preparedness. As a result of these delays, stakeholders may not be able to promptly make improvements in preparedness.

FEMA has also had limited success in ensuring that program requirements for the National Exercise Program have been followed for Principal Level Exercises. HSEEP guidance and the implementation plan state that the results of exercises should be documented and corrective action responsibilities assigned and tracked. Specifically, HSEEP guidance calls for organizations to (1) complete an after-action report with an improvement plan and (2) track corrective actions to ensure that they are implemented. Based on our review of four after-action reports issued by HSC between April 2007 and August 2008, three did not meet program requirements. In one case, the after-action report was drafted but not finalized. In two other cases, the report did not identify officials who were responsible for ensuring corrective actions were implemented. For example, the after-action report for the February 2008 Principal Level Exercise on pandemic influenza stated that the integration of strategic communications and policy remains difficult and should be addressed by an interagency group. However, the after-action report did not identify a department or an agency official who was to be responsible for ensuring that this corrective action was implemented. FEMA officials said they relied on the National Exercise Program Implementation Plan as the guiding policy for HSC’s responsibilities for documenting after-action reports and tracking and resolving corrective actions. However, the plan does not describe a procedure for ensuring that these requirements are met by HSC. Based in part on our review of Principal Level Exercises, FEMA has subsequently taken action to identity officials who are responsible for nearly all of the corrective actions outlined in these after- action reports.

The implementation plan for the National Exercise Program also requires entities responsible for Tier I level exercises, including Principal Level Exercises, to ensure that corrective actions are resolved. However, HSC did not follow the required corrective action process. According to HSC staff, the council does not use the CAP system or another tracking procedure for determining whether corrective actions were implemented. Rather, it delegates the responsibility of taking corrective actions to the appropriate agencies or departments.

FEMA officials agreed that tracking corrective actions for Principal Level Exercises is problematic and impairs their ability to fulfill FEMA’s statutory obligation to track corrective actions from exercises. However, they stated that they do not have the authority to direct HSC or other federal entities to track corrective actions and report this information to FEMA. FEMA officials said that they rely on the National Exercise Program Implementation Plan to set forth HSC’s responsibilities for documenting after-action reports and tracking and resolving corrective actions.

Although the Post-Katrina Act does not give FEMA the authority to compel other federal entities to comply with the objectives of the National Exercise Program, the act places responsibility for implementing the National Exercise Program on FEMA, in coordination with other appropriate federal entities. Therefore, it is incumbent on FEMA to coordinate with HSC and other federal entities to better ensure that FEMA obtains the information it needs to meet its statutory responsibility to track corrective actions. In this regard, the implementation plan for the National Exercise Program does not set forth FEMA’s statutory responsibility to track corrective actions, nor does it require federal departments and agencies to report corrective action information to FEMA. On the contrary, the implementation plan provides that departments and agencies “may submit issues to the DHS Corrective Action Program (DHS CAP) through the web-based DHS CAP system,” but does not instruct them to do so or to otherwise provide FEMA with corrective action tracking information that would enable FEMA to fulfill its statutory responsibilities under the Post-Katrina Act. Therefore, the implementation plan lacks procedures that call on HSC and other federal entities to report corrective action information to FEMA. FEMA’s inability to fully track and analyze areas that need improvement is also due, in part, to its lack of an effective internal controls environment. GAO’s Standards for Internal Controls in the Federal Government state that an effective internal control environment is a key method to help agency managers achieve program objectives and enhance their ability to address weaknesses. The standards state, among other things, that agencies should have policies and procedures for ensuring that the findings of audits and reviews are promptly resolved. The standards also state that internal controls should generally be designed to assure that ongoing monitoring occurs. Developing procedures for working with federal entities, such as HSC, to help ensure that corrective actions are tracked, implemented, and reported to FEMA would strengthen FEMA’s ability to determine emergency management areas that need improvement.

Lessons learned from Hurricane Katrina identified similar concerns with tracking corrective actions from exercises. For example, in February 2006 the White House report on Hurricane Katrina stated that “too often, after- action reports for exercises and real-world incidents highlight the same problems that do not get fixed”. According to the report, DHS should ensure that all federal and state entities are accountable for the timely implementation of remedial actions in response to lessons learned. According to the report, the success of the preparedness system depends, in part, on feedback mechanisms for tracking corrective actions. When federal entities carry out processes that are incompatible with FEMA’s responsibilities for tracking corrective actions, FEMA managers do not have the necessary data to measure progress, identify gaps in preparedness, and track corrective actions—key objectives of the National Exercise Program.

FEMA Does Not Have Procedures to Help Ensure That After-Action Reports from State-Level Exercises Are Properly Prepared and Submitted and That Corrective Actions Are Tracked and Implemented

Similar to the problems we found with Principal Level Exercises, we identified weaknesses in the way in which selected states prepared and submitted after-action reports to FEMA. Among other things, HSEEP requires that exercise program managers prepare after action-reports that include improvement plans to identify corrective actions, track whether the actions were implemented, and continually monitor and review corrective actions as part of an organizational corrective action program. Exercise program managers are to submit these after-action reports to FEMA through its Secure Portal—a FEMA database containing after- action reports. Of the six states we visited, (1) none systematically recorded or submitted after-action reports to FEMA, (2) only one had improvement plans in all of its after-action reports and only one said it had a corrective action tracking program, and (3) none used a capabilities- based approach in all of their exercises. The following are additional details on each of these issues.

HSGP guidance requires exercise program managers to submit after- action reports within 60 days following the completion of an exercise to FEMA through its Secure Portal. Although the portal has been operational for about 5 years, FEMA did not have procedures in place to fully monitor state actions and to ensure that this occurred. While 3 of 44 after-action reports provided by officials from the six states we visited were submitted to FEMA in the requisite area of the portal, these reports all came from one state. The remaining five states did not submit any after-action reports through the portal. Officials from these five states cited technical difficulties, lack of staff resources, or unclear guidance from FEMA as reasons why after-action reports were not submitted to the portal. FEMA is aware that the portal contains incomplete information, noting in its quarterly newsletter that “The Secure Portal serves as the repository for after action reports and improvement plans; however, postings have been inconsistent. At times, it has been difficult to locate After Action Reports as many are posted in draft form and never finalized and posted outside the respective State folder.” In February 2009 FEMA announced, a National Exercise Division Exercise Support System that is an online tool for facilitating exercise planning to replace the Secure Portal as a repository for after-action reports.

HSEEP requires that entities include an improvement plan as part of their after-action reports and that they have a corrective action program. While each of the six states we visited had produced at least one draft after-action report that included an improvement plan, only one state included an improvement plan in all of its reports. Fifteen of the 44 after-action reports we reviewed had an improvement plan. In addition, only one state had a corrective action program that tracked whether corrective actions were implemented. Officials from one state attributed the lack of a corrective action program to competing priorities. Specifically, states are involved in many exercises and officials are more likely to place the priority on designing and conducting of the next exercise than on tracking corrective actions from prior exercises.

Another HSEEP requirement calls for exercises to be designed and conducted using a capability-based approach. Doing so would help FEMA analyze whether gaps in capability have narrowed and improvements in capabilities have occurred from the use of grant funds by states. However, 20 of the 44 after-action reports provided by officials in the six states we visited used target capabilities, while the remaining 24 did not. According to officials from three of the six states we visited, not all exercise participants have a good understanding of target capabilities and how they should be used in the design, conduct, and evaluation of exercises. For example, exercise officials from one state said their state does not use target capabilities because it has its own set of assessment standards.

FEMA’s lack of procedures for monitoring states to ensure compliance with HSEEP requirements contributed to limited adherence to such requirements in the states we visited. As discussed earlier in this report, internal control standards call for (1) an effective internal control environment to help agency managers achieve program objectives and enhance their ability to address weaknesses, (2) agencies to have policies and procedures for ensuring that the findings of audits and reviews are promptly resolved, and (3) internal controls to generally be designed to assure that ongoing monitoring occurs. FEMA officials in the National Preparedness Directorate told us they have a process for monitoring HSEEP compliance by, among other things, having FEMA regional exercise support program managers discuss HSEEP compliance with state exercise program officials at planning conferences or during grant monitoring discussions. While discussing HSEEP requirements at annual conferences may enhance the awareness of state officials about requirements for HSEEP compliance, this does not track compliance. In addition, officials from FEMA’s Grant Programs Directorate said they do not monitor states’ compliance with HSEEP requirements. For example, the grant monitoring reports for the six states we visited did not address whether the states were in compliance with HSEEP requirements. Such reports are based, in part, on a checklist of items that officials use to monitor compliance with grant requirements. However, FEMA’s checklist does not include specific items, such as compliance with HSEEP requirements, as called for by HSGP guidance. States’ noncompliance with HSEEP hinders their ability to systematically track corrective actions and assess capabilities. This in turn impacts FEMA’s ability to measure progress of the National Exercise Program. Having procedures in place to monitor actions by states to ensure compliance with HSEEP requirements would assist FEMA in obtaining more complete data about the results of exercises and corrective actions taken to systematically evaluate readiness through the National Exercise Program, as required by the Post-Katrina Act.

FEMA’s Ability to Design Exercises That Stress the Preparedness System Is Limited

The Post-Katrina Act requires FEMA to stress the preparedness system through the National Exercise Program to evaluate preparedness for a catastrophic event. The National Exercise Program Implementation Plan identifies domestic incident management for catastrophic events as the principal focus of the National Exercise Program. According to the Tier I exercise cycle established in the implementation plan, FEMA plans to test for a catastrophic domestic nonterrorism event in fiscal year 2010. However, FEMA’s ability to meet this testing requirement is limited by three factors: (1) the lack of key planning documents, (2) exercise artificiality, and (3) limited coordination with groups that have expertise in populations with special needs.

First, the effectiveness of exercises is based, in part, on the degree to which plans that define roles and responsibilities have been developed.

The fact that key planning documents for response to a catastrophic incident, such as the supplement to the catastrophic incident annex and regional response plans, have not yet been completed means that the National Exercise Program will have difficulty in designing exercises that test whether the plans are understood and executed effectively by stakeholders.88, 89 As we described earlier in this report, while DHS and FEMA are working on these plans, it is unclear when they will complete the plans. According to the former director of FEMA’s National Preparedness Directorate, FEMA’s ability to design and conduct exercises that evaluate a response to a catastrophic incident is limited by the fact that plans such as those described above have yet to be developed. However, the official indicated that FEMA was taking preliminary actions to build its capacity to conduct such exercises through regional catastrophic planning initiatives.

The supplement to the catastrophic incident annex is required by the Post-Katrina Act. 6 U.S.C. § 319(b)(2)(C). While a version of the supplement was written before the act, an updated supplement has yet to be published.

Regional plans have been drafted; however, according to FEMA officials, the plans are not consistent from one region to another because regions developed plans without any guidance from FEMA headquarters.

Exercise Program’s Implementation Plan, such participation from other federal stakeholders is not required for Tier II exercises such as the one held in May 2008.

Third, another challenge in creating exercises that stress the preparedness system and simulate real-world conditions is finding ways to test response capabilities for populations with special needs. To address some of the problems experienced in Hurricane Katrina in dealing with populations with special needs, such as residents in nursing homes, the Post-Katrina Act, as amended by the 9/11 Act, called on FEMA to design exercises to address the unique requirements of populations with special needs, including the elderly, and to coordinate the National Exercise Program with the National Council on Disability, among other entities. In TOPOFF 4, FEMA integrated specific objectives for special needs populations in the Oregon venue, according to FEMA officials. For example, according to FEMA officials, FEMA utilized special needs actors to enhance realism. However, HSEEP guidance does not address special needs populations. Further, while FEMA has corresponded with the Council on Disability, council officials believe that FEMA could do more to ensure that exercises are designed to address the unique requirements of populations with special needs. For example, council officials stated that the council was not involved in the design and planning for TOPOFF 4 or the May 2008 Tier II exercise. According to officials from the National Preparedness Directorate, it coordinated with FEMA's Special Needs Office to integrate special needs population objectives into exercises. FEMA officials agree that special needs populations should be included in exercises and they said that they will redouble their efforts to do so. However, FEMA officials also noted that some exercises, for example, the National Level Exercise planned in July 2009, may not involve special needs populations because the point of such exercises is to prevent a terrorist attack, rather than to test response and recovery efforts. Enhancing coordination with the National Council on Disability could improve FEMA’s ability to ensure that key issues concerning populations with special needs are addressed in the design and conduct of exercises.

The limitations the National Exercise Program faces in designing approaches that stress the preparedness system highlight the difficulty in validating whether roles and responsibilities are well understood and major gaps in capabilities remain for responding to and recovering from catastrophic events. The 2006 White House report on the federal response to Katrina concluded that the “national preparedness system must be oriented toward greater challenges. We must not shy away from creating scenarios that stress the current system of response to the breaking point……Until we meet the standards set by the most demanding scenarios, we should not consider ourselves adequately prepared.” In 2006, we reported that effective exercises should involve scenarios that stress responders with the highest degree of realism possible, even to the breaking point if possible. Exercises that stress the preparedness system in a realistic way are key to testing the prospective reliability of a response and determining whether plans have accounted for potential breakdowns with relatively greater consequences.

In February 2009, we met with FEMA officials to discuss this issue and they agreed that developing exercises to the point of system failure is a valid objective; however, they described several factors that may limit their ability to do so. For example, FEMA officials told us that exercising to the “breaking point” requires significant resources that under current, and likely future funding streams, are unlikely to be available. We agree with FEMA that these are important considerations; however, these considerations, in part, are addressed through the implementation plan for the National Exercise Program, which describes a 5-year schedule of exercises to give federal departments and agencies lead time to budget for participation in such events.

FEMA’s Data for Measuring the Effectiveness of Its National Exercise Program Is Incomplete

The three databases that FEMA uses to measure the effectiveness and progress of the National Exercise Program have incomplete data. FEMA uses (1) the NEXS system to identify DHS-funded exercises, (2) the FEMA Secure Portal as a repository for DHS funded exercise after-action reports, and (3) the CAP system as a tool for tracking corrective actions. The following provides details on problems with the reliability of each of these databases.

FEMA calls on state entities to use the NEXS system to schedule all exercises, and one of the performance measures that FEMA uses to assess and report on the performance of the National Exercise Program is the number of DHS-funded state exercises that occur per year. However, 26 of the 44 after-action reports we reviewed did not have the exercise entered into the NEXS system. Incomplete NEXS system data limit FEMA’s ability to accurately report on the number of DHS-funded exercises. Furthermore, while FEMA created the NEXS system to schedule, synchronize, and avoid conflicts in all national, federal, state, and local exercises, it cannot do so with incomplete data. When we discussed this problem with state and FEMA officials, they agreed that the NEXS system did not contain a comprehensive list of all state and local exercises supported by HSGP funds. For example, when we asked FEMA for a complete list of all exercises to be conducted under the National Exercise Program, the agency could not produce such a list. Standards for Internal Control in the Federal Government state that program managers need data to determine whether they are meeting their performance targets and that controls should be designed to validate the integrity of organizational performance measures and indicators. FEMA has initiated actions to validate the accuracy of data used in the NEXS system by using training and exercise plan workshops with states to determine what exercises states have scheduled. We agree that the training and exercise plan workshops are a good starting point for verifying the completeness of NEXS; however, when we attended a training and exercise plan workshop, we were told by FEMA officials that not all federal agencies or local entities participated, thus, all exercises were not discussed at the workshop. In addition, FEMA officials recognize that the workshops alone do not ensure that the NEXS database is complete and accurate. In the absence of systematic and comprehensive information on the number of federally funded exercises, FEMA cannot measure its progress in implementing the National Exercise Program.

A second database used by FEMA is the Secure Portal—the primary database that FEMA uses to measure the degree to which states comply with HSEEP. Even though FEMA requires state exercise program managers to place their after-action reports in the Secure Portal when federal grant funds are used to support the exercise, this requirement was not completely met by any of the six states we visited. In addition, although state and FEMA officials agreed that the Secure Portal does not contain all state exercise after-action reports, FEMA uses information from the portal to assess and report on the performance of the National Exercise Program, including the measure that FEMA uses to assess the percent of DHS-funded exercises demonstrating the use of HSEEP guidance. Since the Secure Portal contains incomplete information and neither FEMA nor the six states we visited have controls to ensure that all state-level exercise after- action reports are uploaded to the Secure Portal, this measure may not accurately reflect the percent of DHS-funded exercises that demonstrate the use of HSEEP guidance.

Third, although federal entities involved in Tier 1 exercises are encouraged to use FEMA’s CAP system, it does not contain all corrective actions from such entities. According to FEMA officials, the CAP system was designed to capture all relevant and necessary information related to the implementation of corrective actions. However, the HSC did not use the CAP system to track corrective actions. The problem with having incomplete data in the CAP system is that FEMA uses information from the system to measure the percentage of corrective actions that have been implemented as one of its performance measures. A key reason for this problem is that FEMA has not established procedures to ensure that the information in the CAP system is complete. For example, the implementation plan does not require the nine federal departments and agencies that are signatories to the implementation plan to use the CAP system. Instead, FEMA strongly urges stakeholders to use the system, but the decision to do so is discretionary. FEMA officials cited the tension between requiring entities to use the CAP system and providing enough flexibility to those entities to carry out their programs as a reason for not making the use of the CAP system a requirement. Nonetheless, entities could submit a report to FEMA on the status of their corrective actions resulting from such exercises.

Finally, the CAP system does not include corrective actions from real- world incidents and FEMA has not established requirements or guidelines for agencies to do so. As a result, FEMA is unable to meet Post-Katrina Act requirements for conducting long-term trend analyses of corrective actions that include real-world events. The Presidential Inaugural Ceremony held in Washington D.C. on January 20, 2009, provides an example of the importance of tracking corrective actions for real-world events. During the event, problems with managing crowds prevented a large number of ticket holders from reaching their designated area to observe the inauguration ceremony. According to the Joint Congressional Committee on Inaugural Ceremonies, a complete examination will take place to provide a foundation of lessons learned for future inaugural planners, so that they can avoid similar problems in the future. FEMA officials agreed that the CAP system could be used to track corrective actions from real-world events, and in February 2009 they indicated that developing procedures to do so would aid their ability to conduct long- term trend analyses of real-world events as required by the Post-Katrina Act.

FEMA Has Taken Actions to Develop and Implement a Comprehensive System for Assessing National Preparedness Capabilities, but Faces Challenges in Completing and Implementing the System

FEMA has taken initial actions to collect information on state preparedness capabilities and develop a comprehensive assessment system for assessing capabilities at all levels of government, but faces methodological and coordination challenges in completing the system. Assessing and reporting on national preparedness is a long-standing and complex effort that presents methodological, integration, and coordination challenges. Effectively addressing these challenges requires that FEMA take a measured and planned approach; however, FEMA’s project management plan does not fully identify the numerous program elements and how and when they will be developed and integrated.

FEMA Has Taken Initial Actions to Develop a Comprehensive Assessment System and Collect Information for State and Federal Capability Reporting

The Post-Katrina Act requires that FEMA establish a comprehensive assessment system to assess the nation’s capabilities and overall preparedness for preventing, responding to, and recovering from natural and man-made disasters. The act also requires that FEMA collect information on state capability levels and report on federal preparedness to Congress, including, among other things, the results of the comprehensive assessment system. In response to these requirements, FEMA established guidance for reporting on state preparedness and created the Office of Preparedness Policy, Planning and Analysis (PPPA) to develop and implement a new assessment approach that considers past efforts and integrates ongoing assessment efforts. FEMA plans to integrate the state preparedness reports—along with a variety of existing assessment efforts and data sources—into the new comprehensive system it is establishing. In addition, it is considering the historical experiences and lessons learned from prior assessment efforts in developing the new system.

FEMA has also made progress in collecting information for federal and state reporting. In January 2009, FEMA issued its first federal preparedness report. The Post-Katrina Act requires that FEMA, in coordination with the heads of appropriate federal agencies, submit a federal preparedness report to Congress beginning in October 2007 and annually thereafter, which is to include, among other things, the results of the comprehensive assessment. In addition, states, territories, and the District of Columbia completed and submitted their first state preparedness reports to FEMA in the spring of 2008—a total of 56 reports from the 56 jurisdictions receiving homeland security grant funding. FEMA officials said they prepared summaries of the 56 reports and provided the summaries to FEMA’s regional offices. In addition, in November 2008, FEMA issued guidance for the 2008 state preparedness reports, which grantees are to submit to FEMA by March 2009.

FEMA Faces Methodological and Coordination Challenges in Completing the Comprehensive Assessment System and Reporting on Its Results

FEMA faces methodological challenges with the four assessment systems it plans to use as the basis for the new system and has not determined how to overcome problems faced in historical assessments. The challenges FEMA faces reflect the lack of guidance from PPPA in how the assessment system will comprehensively inform and incorporate feedback from other elements of the National Preparedness System and information from a variety of other data sources. Finally, FEMA faces challenges in coordinating with state, local, and federal stakeholders in developing and implementing the system and reporting on its results.

Methodological Challenges with Using Four Proposed Assessment Approaches

In December 2008, FEMA provided us with a project management plan outlining efforts to establish the comprehensive assessment system by May 2010 to “function as a central repository for national preparedness data.” The system “will integrate data from prior reports and legacy assessment systems.” To establish the system, FEMA plans to administer a Web-based survey to all states and territories in the summer of 2009 to assess capabilities using the 37 target capabilities. FEMA plans to use a Web-based system known as the National Incident Management System Compliance Assessment Support Tool (NIMSCAST) to administer the capability assessment survey. In addition, FEMA noted that NIMSCAST will serve as the technical foundation for the comprehensive assessment system, and that the system is used by all states and territories as well as by 18,000 local and tribal entities, which helps to mitigate challenges FEMA faces in coordinating with stakeholders in developing and implementing the comprehensive assessment system. However, FEMA faces methodological challenges with regard to (1) differences in data available, (2) variations in reporting structures across states, and (3) variations in the level of detail within data sources requiring subjective interpretation, as summarized in table 2 below. Additional information regarding these assessments is outlined in appendix III.

In addition to the methodological challenges in its current approach to assessing capabilities, previous efforts at assessing capabilities experienced challenges and have been discontinued.

The National Preparedness System was discontinued by DHS officials because it was time-consuming and did not produce meaningful data, according to FEMA officials. The system was pilot tested in 10 states and according to budget documentation, FEMA spent nearly $15 million in total for 2006, 2007, and 2008 on the system before it was discontinued.

The Pilot Capability Assessment was labor-intensive and did not generate meaningful data, according to FEMA officials. This assessment, piloted in six states, was intended to measure jurisdictions’ progress in achieving needed target capabilities. Because it was only piloted, FEMA did not generate meaningful preparedness information from the data collected, according to officials.

The Capability Assessment for Readiness, which was proposed as a one-time nationwide assessment of capabilities, lacked controls for validating the accuracy of self-reported assessment data. The assessment was conducted in 1997 but concerns regarding self- reporting and the lack of controls for validating information reported by states limited the reliability and, therefore, the value of the data, according to the DHS Inspector General.

Additional information regarding these efforts is outlined in appendix III.

FEMA Faces Integration Challenges in Completing the Comprehensive Assessment System

FEMA has not established an approach for how information and data from different sources will be integrated into the comprehensive assessment system. FEMA officials have established a charter between the National Preparedness Directorate and Grant Programs Directorate to coordinate preparedness efforts related to the Cost to Capability Initiative and refinement of the target capabilities. FEMA has also begun sharing information between staff involved in developing the assessment system and staff involved in other elements of the National Preparedness System. For example, staff from the National Exercise Program said they shared information on their exercise efforts with staff developing the comprehensive assessment system and FEMA officials said they established a working group of officials from other federal agencies to communicate efforts to develop the comprehensive assessment system. In March 2009, FEMA officials acknowledged that they had not finalized a charter for this working group to outline the specific actions that the working group will undertake to develop the comprehensive assessment system. FEMA explained that this working group will (1) identify existing sources of data related to preparedness plans, organization, equipment, training, and exercises; (2) vet the relevancy of each data source for assessments; (3) identify data gaps and redundancies; and, (4) develop recommendations for streamlining data collection and reporting. However, FEMA has not established an approach for integrating information and data from other stakeholders, including grantees and other FEMA divisions such as the Disaster Operations Directorate.

In October 2008, FEMA officials said they also plan to consider or incorporate into the new system a multitude of other data and analysis sources within and outside of DHS, such as FEMA’s Biannual Strategy Implementation Reports; Homeland Security Grant Program Investment Justification; and Tactical Interoperable Communications Plan Scorecards. In addition, FEMA plans to use the CAP system and LLIS to inform the comprehensive assessment system. In February 2009, FEMA officials further explained that they plan to rely on three indicators of preparedness to develop the comprehensive assessment system: (1) state and federal preparedness reports, which are required to use target capabilities; (2) the results of exercise corrective action findings; and (3) operational plans outlining specific operational requirements for all levels of government. However, FEMA has not established an approach for how the three indicators of preparedness will be collected and developed into reporting mechanisms that meet Post-Katrina Act requirements for the comprehensive assessment system. In its first federal preparedness report, FEMA acknowledged that its efforts to evaluate and improve preparedness are the least mature elements of the national preparedness system because these efforts are composed of a wide range of systems and approaches with varying levels of integration. Given the relative immaturity of FEMA’s evaluation and improvement efforts, without an approach for integrating its comprehensive assessment system efforts, FEMA faces increased risks that inconsistencies may occur or that data and information are not shared, limiting FEMA’s ability to fulfill the requirements of the Post-Katrina Act for developing the system.

FEMA Faces Coordination Challenges in Establishing Quantifiable Metrics and Collecting Information for Implementing the Comprehensive Assessment System

In addition to methodological and coordination challenges in developing and completing the comprehensive assessment system, FEMA faces coordination challenges in establishing quantifiable metrics for target capabilities outlined in the Target Capabilities List. Establishing quantifiable metrics for target capabilities is a prerequisite to developing assessment data that can be compared across all levels of government. At the time of our review, FEMA was in the process of refining the target capabilities to make them more measurable and to provide local and state jurisdictions with additional guidance on the levels of capability they need. FEMA plans to develop quantifiable metrics—or performance objectives— for each of the 37 target capabilities that are to outline specific capability “targets” that jurisdictions, including but not limited to cities, of varying size should strive to meet. FEMA plans to complete quantifiable metrics for all 37 target capabilities by the end of 2010. As of February 2009, FEMA noted that 6 of the 37 capabilities were undergoing stakeholder review and that FEMA planned to develop quantifiable metrics for a total of 12 capabilities during the 2009 calendar year. However, as of March 2008, FEMA had not developed milestones for completing quantifiable metrics for the remaining 25 target capabilities. Cognizant of the fact that there is not a “one size fits all” approach to preparedness, FEMA also plans to develop performance classes for each target capability in order to account for differences in levels of preparedness across jurisdictions of varying size and risk. FEMA plans to incorporate the new performance objectives and performance classes into the comprehensive assessment system, the federal preparedness report, and the guidance for the state preparedness reports, but has not established a time frame for doing this.

FEMA recognizes the need to coordinate with federal, state, and local stakeholders and ensure that their views are effectively integrated in the development of the metrics, but FEMA historically has faced challenges in coordinating with stakeholders. For example, as we reported in June 2008, FEMA’s efforts to coordinate with stakeholders in developing the NRF were inconsistent and needed to be improved. In addition, such coordination can be time consuming. For example, in July 2005 we reported on DHS’s prior effort to develop a tiered system of metrics based on population density and critical infrastructure in order to (1) assign jurisdictions responsibility for developing and maintaining target capability levels and (2) use the metrics to implement a national “balanced investment program” (with the purpose of directing federal preparedness assistance to the highest priority capability gaps) for national preparedness capabilities. DHS scheduled this system to be developed and completed by October 2008. At the time of our review, these efforts to develop quantifiable metrics for target capabilities were not complete, illustrating the fact that developing metrics in coordination with a variety of stakeholders can take longer than anticipated. In developing the quantifiable capability metrics, FEMA officials told us that they plan to conduct extensive coordination with stakeholders that will entail conducting stakeholder workshops in all 10 FEMA regions and coordinating with all federal agencies with lead and supporting responsibility for Emergency Support Function (ESF) activities associated with each of the 37 target capabilities. Officials said they also plan on briefing the National Advisory Council and the National Council on Disability, and soliciting public comment on the draft quantifiable metrics for each target capability.

One of FEMA’s coordination efforts—working with nonfederal stakeholders and federal agencies responsible for ESF activities— illustrates the large number of stakeholders with whom FEMA plans to coordinate in developing quantifiable metrics for the target capabilities. FEMA also plans to post each revised capability to the Federal Register for comment. With respect to federal agency coordination, FEMA plans to coordinate with each federal agency that has lead and supporting responsibility for ESF activities associated with each of the 37 target capabilities in developing quantifiable capability metrics. For example, for the medical surge capability, the Department of Health and Human Services (HHS) is the primary federal agency responsible for coordinating necessary medical surge capabilities needed during a disaster to provide triage and medical care services. In addition to HHS, 15 other federal agencies and the American Red Cross are designated as supporting agencies and organizations for medical surge capabilities. Coordinating with other federal agencies responsible for ESF activities in developing quantifiable capability metrics would likely entail time, effort, and unforeseen risks. For example, in September 2008 we reported on the risks that FEMA faces in coordinating with external stakeholders, namely the American Red Cross, in collecting and integrating preparedness information necessary to develop the comprehensive assessment system. In that report, we recommended that FEMA take steps to better incorporate information from voluntary organizations related to sheltering and feeding capabilities, which are elements of the mass care target capability, and noted that a comprehensive assessment of the nation’s capabilities should account as fully as possible for the voluntary organizations’ capabilities in mass care. FEMA disagreed with the recommendation, noting that it cannot control the resources of nonprofit and private organizations. In response, we (1) noted that taking steps to assess capabilities more fully does not require controlling these resources, but rather cooperatively obtaining and sharing information and (2) reiterated that such efforts are important for assessing the nation’s prevention capabilities and overall preparedness.

FEMA’s efforts to collect information needed to draft and issue the first federal preparedness report, required by the Post-Katrina Act, also reflect the coordination challenges FEMA faces in implementing the comprehensive assessment system. The Post-Katrina Act requires that FEMA, in coordination with the heads of appropriate federal agencies, submit a federal preparedness report to Congress beginning in October 2007 and annually thereafter, which is to include, among other things, the results of the comprehensive assessment. FEMA issued the first federal preparedness report in January 2009. In response to our comment that the draft report had been under review for 8 months between March 2008 and November 2008, FEMA noted that after completing a review of the report by FEMA and DHS, the report was submitted to the Office of Management and Budget to disseminate to all federal departments and agencies for review and comment. Officials explained that in developing the report, they faced challenges in obtaining information and data from federal agencies because of bureaucratic obstacles for collecting information and also faced challenges in analyzing information from multiple sources. FEMA officials said they may develop a National Preparedness Report to combine two Post-Katrina Act reporting requirements—the requirement for an annual federal preparedness report and an annual catastrophic resources report—and to include information from state preparedness reports as part of this consolidated report, which they tentatively plan to issue in the Spring of 2009.

FEMA Has Not Developed an Approach and Assessed Risks for Completing and Implementing the Comprehensive Assessment System as Part of Its Project Management Plan

Despite the methodological and coordination challenges associated with developing a new comprehensive assessment system and establishing related quantifiable metrics for target capabilities, FEMA has not developed an approach that addresses program risks as part of its project management plan for how it will develop the comprehensive assessment system. While FEMA has developed a project management plan for completing the comprehensive assessment system by 2010, the lack of (1) milestones for establishing quantifiable metrics for all 37 target capabilities and (2) specific actions for how FEMA will integrate preparedness information to develop the system, coupled with the (3) the lack of risk assessment information for the system raises questions about FEMA’s ability to establish the system in accordance with its anticipated 2010 completion date. FEMA has described several steps for completing an effective comprehensive assessment system that include developing methodologies to translate the information from the assessments FEMA has identified into a target capabilities-based framework and integrating necessary preparedness information from disparate and not necessarily comparable sources such as state preparedness reports. FEMA also plans to coordinate with relevant stakeholders to refine the target capabilities.

However, FEMA’s plan does not outline specific actions it plans to take to do so.

Certain factors, such as challenges in coordinating with stakeholders or difficulties in obtaining necessary data, could affect FEMA’s ability to implement the comprehensive assessment system. Best practices for project management established by the Project Management Institute state that managing a project involves project risk management, which serves to increase the probability and impact of positive events, and decrease the probability and impact of events adverse to the project. Project risk management entails determining which risks might affect a project, prioritizing risks for further analysis by assessing their probability of occurrence, and developing actions to reduce threats to the project. Other practices include (1) establishing clear and achievable objectives; (2) balancing the competing demands for quality, scope, time, and cost; (3) adapting the specifications, plans, and approach to the different concerns and expectations of the various stakeholders involved in the project; and (4) developing milestone dates to identify points throughout the project to reassess efforts underway to determine whether project changes are necessary. FEMA has demonstrated its awareness of the value of these practices. For example, in planning another project—an effort to transition the FEMA Secure Portal to an alternate information technology platform known as the National Exercise Division Exercise Support System —FEMA identified key elements such as phases, milestones, and risks that could affect the project goals.

Furthermore, a risk assessment could help FEMA define the specific actions to take to complete the comprehensive assessment system, anticipate potential delays in completing its efforts to refine the target capabilities by 2010, and deal with the associated risks in its efforts to do so, such as the time it takes to coordinate with stakeholders. Information from a risk assessment could also enhance FEMA’s ability to coordinate with federal agencies to obtain preparedness information needed to produce a timely annual federal preparedness report and catastrophic resources report. Until FEMA assesses ways to mitigate the risks associated with its capability assessment efforts, it will be difficult for FEMA to provide reasonable assurance that it can produce a comprehensive assessment system that (1) fulfills the requirements of the Post-Katrina Act and in the long term, (2) informs decisions related to improving national preparedness.

FEMA’s Implementation of the National Preparedness System Would Be Enhanced by a Strategic Plan That Includes Desirable Characteristics

FEMA’s National Preparedness Directorate does not have a strategic plan for implementing the national preparedness system. The complexity and difficulty of implementing the national preparedness system, which we describe earlier in this report, underscore the importance of strategic planning. The six desirable characteristics of a national strategy can help the National Preparedness Directorate in developing a strategic plan.

FEMA’s National Preparedness Directorate Does Not Have a Strategic Plan for Implementing the National Preparedness System

While FEMA has recognized that its components need to develop strategic plans that detail program goals, objectives, and strategies, FEMA’s National Preparedness Directorate (Preparedness Directorate) has not yet developed such a plan for the national preparedness system. In January 2008, FEMA issued its agencywide strategic plan, which set a common direction for its components in carrying out their responsibilities in preparedness, response, and recovery programs. While a Preparedness Directorate official acknowledged that the Preparedness Directorate does not have a strategic plan (this is the responsibility of FEMA’s Office of Preparedness Policy, Planning and Analysis), the official said the Post-Katrina Act provides a roadmap that contains the preparedness strategy and FEMA uses an annual operating plan (in draft form at the time of our review) to guide the directorate’s approach for implementing the national preparedness system.

Although the Post-Katrina Act and the directorate’s draft annual operating plan outline certain elements of a strategy, such as the directorate’s vision, mission, and goals, they do not include several other desirable characteristics of a strategic plan, such as a discussion of how the directorate will (1) measure its progress in developing the national preparedness system, (2) address risk as it relates to preparedness activities, (3) coordinate with its preparedness stakeholders in developing and carrying out the various elements of the national preparedness system, and (4) integrate the elements of the national preparedness system. For example, FEMA has not included information on performance measures for meeting one of the objectives outlined in the operating plan—to support an integrated planning system for the federal preparedness-related agencies that links to regional, state, and local planning activities. The operating plan also does not define the problem or assess the risks that FEMA’s national preparedness program faces. Specifically, it does not describe the threats, vulnerabilities, and consequences of a major homeland security incident or what FEMA’s approach will be for addressing risk through its national preparedness system activities. While the draft operating plan identifies subcomponents in the Preparedness Directorate that will be responsible for carrying out segments of the national preparedness system, it does not discuss the roles and responsibilities of preparedness stakeholders, the coordination that will occur between them, or how the four elements of the national preparedness system will be integrated.

The Post-Katrina Act calls on FEMA to develop and coordinate the implementation of a strategy for preparedness, and the complexity and difficulty of developing a national preparedness system underscore the importance of strategic planning. An important element of strategic planning is that it presents an integrated system of high-level decisions that are reached through a formal, visible process. The resulting strategy is thus an effective tool with which to communicate the mission and direction to preparedness stakeholders. The conditions we describe earlier in this report—such as incomplete plans on roles and responsibilities, unresolved corrective actions from exercises, and potential difficulties and historical delays in capability assessments—show that FEMA faces significant challenges in developing the key elements of the national preparedness system.

The Desirable Characteristics of a National Strategy Can Help FEMA’s National Preparedness Directorate in Developing a Strategic Plan

In 2004, we identified six desirable characteristics of an effective national strategy that help achieve strategy success. These characteristics are summarized in table 3.

We believe these characteristics can assist responsible parties, such as FEMA, in further developing and implementing national strategies as well as enhancing these strategies’ usefulness for policy decisions to help achieve program results and accountability. The characteristics are a starting point for developing a strategic plan. However, we believe that an approach incorporating the substance of these characteristics is likely to increase success in strategy implementation. The following describes how each of these characteristics applies to the work of FEMA’s National Preparedness Directorate.

Purpose, scope, and methodology: National preparedness is an important part of homeland security efforts outlined in legislation, presidential directives, and policy documents. The National Strategy for Homeland Security recognized the importance of fostering a Culture of Preparedness that permeates all levels of our society, including all preparedness stakeholders. In summarizing lessons learned from Hurricane Katrina, the White House report made over 100 recommendations and concluded that an immediate priority for correcting the shortfalls in the federal response to Hurricane Katrina was to define and implement a comprehensive national preparedness system. We believe a strategic plan for implementing the national preparedness system that includes a clearly stated purpose, scope, and methodology could help the Preparedness Directorate convey to preparedness stakeholders the importance of integrating the multiple elements of the national preparedness system and interagency coordination.

Problem definition and risk assessment: As shown by 9/11 and Hurricane Katrina, the nation faces risks from terrorist attacks and man- made and natural disasters. These threats may vary between localities and regions, but responders are to be able to effectively work together in a common language on operational tasks when required. According to the National Preparedness Guidelines, responders must identify and assess risk to ensure the necessary capabilities are available for selecting the appropriate response. In addition, understanding risk involves assessing what vulnerabilities and weaknesses require further attention. The lessons learned from Hurricane Katrina show that federal agencies were not prepared for a catastrophic disaster. Confusion by emergency responders over their roles and responsibilities was widespread and resulted in a slow or fragmented response. To improve response and recovery to all hazards including a catastrophic disaster, the Post-Katrina Act called on FEMA to develop the national preparedness system.

Goals, subordinate objectives, activities, and performance measures: The cultural shift of the preparedness community from a response and recovery strategy to a proactive preparedness strategy emphasizes the importance of a strategic plan that includes clear goals, objectives, activities, and measures. Identifying performance measures for the various components of the national preparedness system, which is also a requirement under the Post-Katrina Act, will help policymakers determine what progress has been made and what remains to be done, especially as it relates to preparedness for a catastrophic disaster. A strategic plan that outlines an overarching goal, subordinate objectives, activities, and performance measures for the various components of the national preparedness system would help FEMA prioritize future efforts and allow decision makers to measure progress.

Resources, investments, and risk management: Preparedness agencies are to manage their likely risks and direct finite resources to the most urgent needs. The national preparedness system helps inform decision makers in federal and state agencies on their use of resources relative to their level of capabilities achievement. Different states and areas face different risks, and thus should have different capabilities to mitigate those risks. As we reported in March 2008, although DHS has taken some steps to establish goals, gather information, and measure progress, its monitoring of homeland security grant expenditures does not provide a means to measure the achievement of desired program outcomes. FEMA’s current efforts do not provide information on the effectiveness of those funds in improving the nation’s capabilities or reducing risk. The National Strategy for Homeland Security describes how resources and risk management must be addressed in a comprehensive approach. For example, the strategy states that “We must apply a risk-based framework across all homeland security efforts in order to identify and assess potential hazards, determine what levels of relative risk are acceptable, and prioritize and allocate resources among all homeland security partners, both public and private, to prevent, protect against, and respond to and recover from all manner of incidents.” A strategic plan that outlines resources, investments, and risk management would help FEMA coordinate a prioritized approach.

Organizational roles, responsibilities, and coordination: Achieving national preparedness, especially for catastrophic incidents, requires sharing responsibility horizontally with other federal departments and agencies. It also requires a robust vertical integration of the federal, state, local, and tribal governments, as well as private entities. FEMA’s Preparedness Directorate faces the challenge of aligning operations of the nation’s preparedness stakeholders to coordinate activities and plans to implement a national preparedness program capable of dealing with catastrophic incidents. A key part of a national preparedness strategic plan would be the clear delineation of organizations and their roles and responsibilities, as well as processes to coordinate their responsibilities.

Integration and implementation: A national preparedness strategic plan would help describe how preparedness agencies at all government levels and sectors will integrate their various standards, policies, and procedures into the national preparedness system. Plans describing how to integrate and implement the various elements of the national preparedness system would help FEMA inform emergency managers, first responders, and decision makers on how the individual elements of the national preparedness system will improve capabilities, training, and plans for all hazards, including catastrophic disasters. A strategic plan to implement the national preparedness system would enable FEMA’s Preparedness Directorate to improve its likelihood of achieving its vision, evaluating progress, and ensuring accountability of federal agencies and other organizations in aligning their efforts to develop and improve the national preparedness system. While it may be impossible to have absolute compatibility because of the many public and private organizations involved, the danger in organizations using different methods or systems, without some overall guidance to assure consistent application of approaches, is that the elements of the national preparedness system will have little ability to inform one another. More importantly, these systems may produce unreliable or incomplete data on how to improve programs related to response and recovery. FEMA plays a crucial role in this regard through its statutory responsibility of carrying out the Post-Katrina Act requirements for the national preparedness system.

Conclusions

The nation looks to FEMA for leadership to ensure that stakeholders involved in preparedness activities can effectively provide a coordinated response to man-made or natural disasters. The nation’s experiences after the events of 9/11 and the 2005 hurricane season dramatically demonstrated our emergency preparedness capabilities and where they were lacking. The Post-Katrina Act’s centralization of responsibility in FEMA for exercises and the other primary activities that form the national preparedness system provides an unprecedented opportunity for comprehensive integration and coordination. While FEMA has made progress in implementing each of these interdependent and essential preparedness activities, it is difficult to measure this progress. FEMA lacks a comprehensive approach to managing the development of policies and plans and overseeing the National Exercise Program. Additionally, FEMA has not established a clearly defined course of action to assess capabilities based on quantifiable metrics. Finally, FEMA has not established a strategic plan for integrating these elements of the national preparedness system. These conditions show that much remains to be done. In the short term, progress is heavily dependent on continuing to improve basic policies and procedures, management tools, and project plans for key elements of the system. Developing each element of the system is undoubtedly a complex task, but progress has to be built on these incremental but critical steps. In the long term, progress will be increasingly dependent on how well FEMA coordinates with the thousands of stakeholders in the system and the degree to which it can integrate the plans, exercises, and assessments into a cohesive approach that improves national preparedness. This need centers attention on leadership and guidance from the Preparedness Directorate and success will depend on linking the various elements of the system and showing how data and information from the system will inform program and budget decisions related to improving preparedness.

A complete, integrated set of national preparedness policies and plans that defines stakeholders’ roles and responsibilities at all levels is needed to ensure that federal, state, and local resources are invested in the most effective exercises. As one program official told us, “If an exercise is testing an inadequate plan, then the exercise is just an experiment….” Until all national preparedness policies are developed and operational plans are created or revised to reflect changes in the roles and responsibilities of key stakeholders, FEMA’s ability to update requisite training to prepare officials responsible for fulfilling these roles and reflect the preparedness lessons of the unprecedented disasters of the last decade in exercises and real-world response will be limited. Without a program management plan, FEMA cannot effectively ensure, in coordination with DHS and other federal entities, that it will complete and integrate key policies and plans with each other and the national preparedness system as envisioned by law and presidential directive.

In implementing the latest iteration of the National Exercise Program, FEMA has issued guidance and requirements for exercise design, execution, evaluation, and corrective action resolution. However, federal and state exercise officials have not yet fully embraced the essential program components. In addition, opportunities to make exercises as realistic as possible by coordinating more fully with all preparedness stakeholders, including the National Council on Disability, and translating the experiences of real-world incidents into corrective actions could further enhance the value of the exercise program. Establishing policies and procedures that detail how FEMA would work with federal entities as well as monitor states’ compliance in implementing the program would help stakeholders meet program requirements and FEMA develop complete and accurate information on program implementation. More importantly, these key program controls, once more systematically established and applied, will enhance FEMA’s ability to assess the extent to which corrective actions have been implemented and, ultimately, describe strengths and weaknesses in the nation’s preparedness capabilities.

Because a comprehensive assessment of national preparedness capabilities is a monumental task, it is understandable that FEMA’s efforts to develop and implement an assessment approach have been underway for more than a decade. Program officials with responsibility for this most recent effort have recognized the need for a comprehensive set of metrics to identify needed capabilities in equipment, personnel, skills, or processes and to prioritize national investments in preparedness. Given the complexity of this effort, they would benefit from a clear roadmap that details their analytical approach for integrating disparate information sources, identifies associated program risks, establishes more specific milestones to help avoid unexpected setbacks, and provides a basis for assessing program progress and making revisions, if needed, to the agency’s implementation plans. Such mitigation efforts would not eliminate the risks associated with the development of the comprehensive assessment system and target capabilities metrics, but they would provide a basis for holding officials responsible for timely and quality results and hold them harmless for unavoidable or unforeseen events that could delay their efforts.

Finally, effective coordination, integration, and implementation of these elements of the national preparedness system require the combined contributions of a broad range of federal, state, and local stakeholders. FEMA has started this integration effort and has had some success in issuing guidelines and requirements that seek greater uniformity of effort. But our work shows that issuing guidelines alone does not assure consistent application across organizations. While it may be impossible to have absolute compatibility because of the many public and private organizations involved in preparedness, the danger in organizations using different methods or systems—without some overall guidance, direction, and controls in place to assure consistent application of preparedness approaches—is that the elements of the preparedness system will have little ability to inform one another. Perhaps, more important, these systems may produce unreliable or incomplete data on how to allocate resources or to improve programs related to response and recovery, especially with respect to catastrophic incidents. In this regard, FEMA officials have noted that their authority is limited to coordinating with, but not directing, other federal agencies. This condition highlights the importance of developing a strategic approach that leads to partnerships with stakeholders whose cooperation is necessary for developing the preparedness system. Making progress with regard to this challenge is a necessary step to assessing our nation’s capabilities and dealing with gaps in preparedness. The scope and breadth of this critical national effort suggests that an explicit description and elaboration of the elements of the system and the level of effort associated with its effective application could enhance stakeholder acceptance and participation. In addition, defining the end state of the preparedness system will help translate requirements from presidential directives and the Post-Katrina Act into measurable steps for achieving an integrated national system. Developing goals and metrics to measure progress towards achieving an integrated system will help FEMA prioritize actions, requirements, and national investments in preparedness. A strategic plan for the National Preparedness Directorate that describes how it will approach these challenges and mitigate these weaknesses would help FEMA partner with the many organizations whose cooperation and resources are necessary for success.

Recommendations for Executive Action

To ensure that key elements of the national preparedness system are developed in a timely and integrated fashion, we recommend that the Administrator of the Federal Emergency Management Agency take the following 11 actions: Direct the Disaster Operations Directorate and the National Preparedness Directorate to improve their approach to developing policies and plans that define roles and responsibilities and planning processes.

Develop a program management plan, in coordination with DHS and other federal entities, to ensure the completion of the key national preparedness policies and plans called for in legislation, presidential directives, and existing policy and doctrine, to define roles and responsibilities and planning processes, as well as to fully integrate such policies and plans into other elements of the national preparedness system. The program management plan, among other things, should: identify the specific schedule of activities that needs to be performed to complete, and identify dependencies among, all policy and planning development and integration activities; identify the type and quantities of resources required to perform, and the schedule for completing, all policy and planning development and integration activities; analyze activity sequences, durations (including the time required to partner and coordinate on an interagency basis with other federal entities), resource requirements, and schedule constraints to create and update the individual policy and plan development project schedules; and control for changes to the project schedules precipitated by outside forces.

When outstanding policies and plans are completed, integrate them into training and exercise efforts to ensure that roles and responsibilities are fully communicated and fully understood by emergency response stakeholders.

Direct the National Exercise Division to improve its implementation of statutory and program requirements.

Coordinate with the Department of Homeland Security to develop policies and procedures for issuing after-action reports for National Level Exercises (i.e., TOPOFF) in 6 months or less, as required by the implementation plan for the National Exercise Program.

Collaborate with the Homeland Security Council to establish policies and procedures for documenting corrective actions from Principal Level Exercises that are consistent with HSEEP guidance and the implementation plan for the National Exercise Program.

Collaborate with the Homeland Security Council to provide FEMA with the information it needs from past principal level exercises to enable it to conduct remedial action tracking and long-term trend analysis, as required by the Post-Katrina Act.

Ensure compliance by states that receive grant funds with HSEEP requirements by revising FEMA’s grant monitoring guidance, for example by including a checklist of specific HSEEP requirements for state validation and certification.

Involve the National Council on Disability on committees involved in the design and execution of national level exercises, especially on issues related to populations with special needs.

Develop internal control policies and procedures that validate the completeness and accuracy of data used to measure program performance. Such procedures could involve checking whether states and federal agencies are providing data and information needed to measure the performance of the program.

Revise the National Exercise Program Implementation Plan to require the use of FEMA’s Corrective Action Program for all federal exercises that involve interagency testing of roles and responsibilities or require that federal agencies submit a report to FEMA on the status of their corrective actions resulting from such exercises.

Develop procedures for including “lessons learned” from real-world incidents in the Corrective Action Program system.

Direct the Office of Preparedness Policy, Planning, and Analysis to improve its approach for developing a comprehensive assessment system.

Enhance its project management plan to include milestone dates, an assessment of risk, and related mitigation strategies for (1) comprehensively collecting and reporting on disparate information sources, (2) developing quantifiable metrics for target capabilities that are to be used to collect and report preparedness information, and (3) reporting on the results of preparedness assessments to help inform homeland security resource allocation decisions.

Direct the National Preparedness Directorate to take a more strategic approach to developing the national preparedness system.

Develop a strategic plan for implementing the national preparedness system that includes the key characteristics of a strategic plan, including coordination, integration, and implementation approaches.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for comment. In commenting on our draft report, DHS noted that while it may not agree with all the assertions in the report, it generally agreed with our recommendations. DHS stated that GAO’s recommendations provide a useful methodology and sound counsel for revision of FEMA’s current portfolio of national preparedness policy, plans, protocols, and procedures. Specifically, DHS stated that FEMA has already made significant inroads in each aspect of the GAO recommended characteristics for sound strategic planning.

DHS also expressed concern that the report suggests that DHS/FEMA should hold other federal agencies and departments or state, local, or tribal governments accountable for compliance with program requirements, while also recognizing that FEMA did not generally have the explicit authority to compel compliance. The Post-Katrina Act designates FEMA as the federal leader and coordinator for developing and implementing the national preparedness system. We recognize that FEMA’s authority is generally to coordinate, guide, and support, rather than direct, and that collaboration is an essential element of FEMA’s efforts. At the same time, we believe that FEMA’s expanded leadership role under the Post-Katrina Act provides FEMA opportunities for and a responsibility to further develop its relationships with national preparedness stakeholders at the local, state, and federal levels and to instill a shared sense of responsibility and accountability on the part of all stakeholders for the successful development and implementation of the national preparedness system. Several of our recommendations aim to enhance such collaboration and cooperation.

DHS also provided technical comments, which we incorporated into the report as appropriate. Appendix V contains written comments from DHS.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this report. At that time we will send copies of this report to the Secretary of Homeland Security, the Director of the Office of Management and Budget, and interested congressional committees. We will also make copies available to others on request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-8757 or e-mail at jenkinswo@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

This review examined key elements of the national preparedness system, including the National Exercise Program. Specifically, our reporting objectives were to review the extent to which the Federal Emergency Management Agency (FEMA) has: 1. developed policies and plans that define roles and responsibilities and planning processes for national preparedness; 2. taken actions since 2007 to implement the National Exercise Program and track corrective actions at the federal and state levels and what challenges remain; 3. made progress in conducting a nationwide capabilities-based assessment, including developing required preparedness reports, and what issues, if any, it faces in completing the system; and, 4. developed a strategic plan for implementing the national preparedness system.

To address these objectives, we analyzed information and data on FEMA’s policies and plans for preparedness, the National Exercise Program, its approach for developing a comprehensive system for assessing nationwide capabilities, and its strategy for integrating elements of the preparedness system. GAO explored the option of selecting exercises to review, but there is no national database that captures all exercises conducted using Homeland Security Grant Program funds. Therefore we selected six states—California, Georgia, Illinois, New York, Texas, and Washington— that provide examples of how exercises are planned and conducted and visited these six states. While we cannot generalize our work from these visits to all states, we chose these locations to provide examples of the way in which states carry out their exercise and preparedness programs. In selecting these states, we considered factors such as states’ participation in national-level exercises; states located in different geographic locations, such as those in hurricane-prone regions; and states with varying percentages of homeland security grant funding planned to support exercises. At each location, we interviewed staff in FEMA’s regional offices responsible for regional preparedness activities. We interviewed state and local officials on their progress and challenges in carrying out preparedness activities, including exercises and assessments of capability. We analyzed key legislation such as the Post-Katrina Act and the Implementing Recommendations of the 9/11 Commission Act of 2007 (9/11 Act) as well as presidential directives related to preparedness efforts.

We also interviewed FEMA officials responsible for preparedness programs to learn more about the actions they had taken and planned to take related to preparedness efforts and compared FEMA’s policies and procedures with criteria in GAO’s standards for internal control in the federal government.

To analyze the extent to which policies and plans have been developed to define roles and responsibilities and planning processes for national preparedness, we analyzed key legislation, presidential directives, and DHS- and FEMA-issued policies that identify required preparedness policies and plans to define roles and responsibilities for emergency response as well as establish guidance for planning processes for developing emergency response plans. We identified the resulting policies that define roles and responsibilities and that form the basis of the national preparedness system, including the National Strategy for Homeland Security, the National Response Framework (NRF), the National Preparedness Guidelines, and the National Incident Management System. In addition, we identified related policies that supplement these documents, such as guidance for Joint Field Office operations. We also identified policies that define planning processes for developing emergency plans, such as the draft Integrated Planning System (IPS) and FEMA-issued interim and final Comprehensive Preparedness Guides for nonfederal planning efforts. We identified plans developed using planning processes that further define and operationalize roles and responsibilities identified in existing policies. These plans include the incident annexes and the incident annex supplements for the NRF, FEMA’s Pre-Scripted Mission Assignment Catalog, plans being developed as part of FEMA’s Catastrophic Disaster Planning Initiative, as well as plans called for by HSPD 8 Annex 1 that are to be developed using IPS. To identify lessons learned and corrective actions related to roles and responsibilities from federal emergency response exercises, we summarized lessons learned from after-action reports for Tier I and II (or equivalent) exercises from 2005 through 2008. The exercises that comprised this data set were identified by FEMA officials as well as counsel to the White House Homeland Security Council. This analysis was conducted to determine whether the exercises revealed unclear or conflicting roles and responsibilities between federal departments and agencies and if additional policies and plans were needed. We also interviewed officials from DHS’s Office of Operations Coordination and FEMA’s National Preparedness Directorate and Disaster Operations Directorate to obtain information on the status of efforts to develop and issue required preparedness policies and plans, including any existing program or project management plans and related issuance schedules. We compared policies and plans that have been published in a final form, versus released in interim or draft formats or that have not yet been developed, to determine the issuance status (completed, partially completed, or incomplete) of these policies and plans. To identify best practices for program management, such as steps for how a program is to be executed, monitored, and controlled, we reviewed the Project Management Institute’s The Standard on Program Management. Finally, we reviewed our prior reports on FEMA’s preparedness programs and planning efforts, as well as prior DHS, White House, and congressional reports on the lessons learned from the response to Hurricane Katrina in 2005.

To assess the extent to which FEMA has taken actions since 2007 to implement a National Exercise Program and track corrective actions, we observed portions of two National Exercise Program exercises (TOPOFF 4 and the May 2008 Tier II exercise). Specifically, during TOPOFF 4, we discussed exercise implementation with federal, state, and local officials and observed FEMA’s exercise management efforts at the TOPOFF 4 Master Control Cell in Springfield, Virginia; the TOPOFF 4 Long-Term Recovery Tabletop Exercise in Washington, D.C.; and exercise implementation in Portland, Oregon. During the May 2008 Tier II exercise, we observed exercise implementation in Mount Weather and Suffolk, Virginia and in Blaine, Washington and discussed the exercise with participating federal, state, and local officials. We evaluated key program documents, such as the implementation plan for the National Exercise Program and the Homeland Security Exercise and Evaluation Program (HSEEP)—FEMA’s guidance for carrying out exercises in accordance with the Post-Katrina Act and the 9/11 Act and data on the program’s performance measures. We reviewed actions taken by FEMA since 2007 because the National Exercise Program Charter was established in January 2007 and the implementation plan was issued in April 2007. We examined after-action reports for Principal Level Exercises-–exercises which involve senior federal officials, such as Deputy Secretaries of departments or agencies-–that were issued from April 2007 through August 2008 to determine whether the Homeland Security Council developed after-action reports. We also interviewed Homeland Security Council staff and the Associate Counsel to the President on the role and responsibility of the council for systematically tracking and implementing corrective actions resulting from Principal Level Exercises because these staff were responsible for summarizing corrective actions for Principal Level Exercises. We also reviewed after-action reports that were provided to us by the six states we visited for exercises conducted from June 2007 through September 2008 that used Homeland Security Grant Program funds, in order to determine how well these states were complying with HSEEP and grant guidance. To determine if FEMA is conducting monitoring and oversight of Homeland Security Grant Program recipients, we reviewed grant monitoring reports for the six states we visited. For information that would provide a broader perspective on FEMA’s efforts, we examined several FEMA databases, including the FEMA Secure Portal—the FEMA repository of after-action reports; the National Exercise Schedule (NEXS) system—a scheduling system for all exercises; and the Corrective Action Program (CAP) system—which is designed for tracking capability-based improvement plans entered by federal, state, and local exercise participants. We assessed the reliability of the FEMA Secure Portal and NEXS databases by checking the systems to determine if known exercises identified through after-action reports produced by states were included in these systems, and by interviewing FEMA and state officials responsible for the data. In addition, we assessed the reliability of the CAP system by interviewing FEMA and Homeland Security Council officials responsible for the data. We concluded the data in the FEMA Secure Portal, the NEXS system, and the CAP system were not reliable for use in this report because these databases lacked complete information related to after-action reports, scheduled exercises, and corrective actions and FEMA does not have procedures in place to ensure that required data are collected consistently to populate these databases.

To determine the extent to which FEMA has made progress and issues FEMA has encountered in conducting a nationwide capabilities-based assessment and developing required preparedness reports and any issues it faces in completing the system, we (1) analyzed FEMA’s plans and schedules for developing the comprehensive assessment system and performance objectives for measuring capabilities, including assessment efforts initiated by FEMA’s National Preparedness, Disaster Operations, and Grant Programs Directorates, and (2) interviewed FEMA staff responsible for these efforts. We also reviewed assessments previously conducted by DHS and FEMA to evaluate historical efforts to assess capabilities. To assess FEMA’s efforts to establish quantifiable metrics for target capabilities, we analyzed preliminary performance objectives for two target capabilities that FEMA had completed, and interviewed headquarters staff responsible for these efforts. We also reviewed information pertinent to FEMA’s assessment approach, including the Federal Preparedness Report (issued by FEMA in January 2009) and State Preparedness Reports for the 2007 reporting year for the six states we visited (reports for 2008 were due to FEMA after this report was finalized for publication). To assess the comparability of information contained in the six State Preparedness Reports, we selected one target capability— mass prophylaxis—outlined in the Target Capabilities List, and reviewed a performance measure associated with this capability for providing initial prophylaxis (an action taken to prevent a disease or a health problem) within 48 hours of a state/local decision to provide prophylaxis. To identify best practices for project management, such as steps for how a project is to be executed, monitored, and controlled, we reviewed the Project Management Institute’s A Guide to the Project Management Body of Knowledge (PMBOK) and compared FEMA’s efforts for developing and implementing the comprehensive assessment system to the best practices developed by the institute. Finally, we reviewed our prior GAO reports on FEMA’s preparedness programs and exercise efforts.

To determine the extent to which FEMA’s National Preparedness Directorate has developed a strategic plan that implements the national preparedness system, we interviewed FEMA National Preparedness Directorate officials on strategic planning and policy and procedures for the National Preparedness System. We analyzed the Post-Katrina Act and key FEMA documents—including the agencywide Strategic Plan for Fiscal Years 2008 through 2013, the Grants Program Directorate Strategic Plan, and the draft annual National Preparedness Directorate Operating Plan— to determine strategy-related requirements. Additionally we reviewed DHS and DHS Office of Inspector General reports on national preparedness. To determine the elements that comprise a strategic plan, we examined our prior reports on the desirable characteristics of effective national strategies and compared them with FEMA’s current approach for developing a National Preparedness System. Finally, we compared the desirable characteristics to the work of the National Preparedness Directorate as described in the Homeland Security Council’s National Strategy for Homeland Security, DHS’s National Preparedness Guidelines, and the Post-Katrina Act.

We conducted this performance audit from January 2008 through April 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Overview of Policies and Plans to Define Roles and Responsibilities and Planning Processes for Developing Emergency Plans

In September 2006, we reported that in preparing for, responding to, and recovering from any disaster, the legal authorities, roles and responsibilities, and lines of authority for the preparation and response at all levels for government and nongovernmental entities must be clearly defined, effectively communicated, and well understood in order to facilitate rapid and effective decision making. National preparedness policies and plans identify these legal authorities, roles and responsibilities, and lines of authority for response activities and serve to communicate this information to emergency response stakeholders, and, in conjunction with training, are the basis for ensuring that the information is well understood. Effective and efficient disaster management relies on the thorough integration of these emergency response policies and plans. Because emergency response activities entail large numbers of stakeholders who need to be able to respond to an incident in a coordinated and integrated manner, it is essential that roles and responsibilities are defined, communicated, and understood prior to a real-world incident response. An example of the range of stakeholders involved in such response activities is illustrated by figure 6.

The wide range of emergency response stakeholders depicted in figure 6, among others, are to be organized by the roles and responsibilities defined in policies and plans that are designed to facilitate an effective response to an incident, be it man-made (e.g., terrorism) or a natural disaster. Policies that broadly define roles and responsibilities are operationalized by the development of plans that provide greater levels of detail. These detailed plans are to be developed using the planning processes that are discussed and established in federal policies. The range and relative relationships of the policies that define roles and responsibilities and planning processes for developing emergency plans are illustrated in figure 7. Table 4, which follows figure 7, describes each of the policies presented in the graphic.

Among the 50 policies that define roles and responsibilities or planning processes, 46 define roles and responsibilities and 4 define planning processes for developing emergency plans. Table 4 provides brief descriptions and the status of the policies depicted in figure 7 that define roles and responsibilities. Of the 46 policies presented in table 4, 40 have been completed and 6 are incomplete. a standardized federal planning process (the Integrated Planning System); resourced operational and tactical planning capabilities at each federal department and agency with a role in homeland security; strategic guidance statements, strategic plans, concepts of operations, operations plans, and, as appropriate, tactical plans for each National Planning Scenario; and a system for integrating plans among all levels of government.

It also calls for the development of the National Homeland Security Plan.

Policy and doctrine (Issued by DHS or FEMA)

The NSHS is to guide, organize, and unify homeland security efforts by providing a common framework for the prevention of terrorist attacks; protection of people, critical infrastructure, and key resources; and response to and recovery from man-made and natural disasters. It calls for homeland security management through a continuous, mutually reinforcing cycle of four activity phases: (July 2002; revised October 2007)

1. overarching homeland security guidance grounded in clearly articulated and up-to-date homeland and relevant national security policies, with coordinated supporting strategies, doctrine, and planning guidance flowing from and fully synchronizing with these policies; 2. a deliberate and dynamic system that translates policies, strategies, doctrine, and planning guidance into a family of strategic, operational, and tactical plans; 3. the execution of operational and tactical-level plans; and 4. continual assessment and evaluation of both operations and exercises.

NHSP, as called for under HSPD 8 Annex 1, is to be an overarching strategic plan to guide national efforts to execute the National Strategy for Homeland Security. It is intended to: facilitate federal homeland security coordination, (release schedule undetermined) define roles and responsibilities for preventing, protecting against, responding to, and recovering from man-made and natural disasters.

It was to be submitted to the President for approval within 120 days of the approval of HSPD 8 Annex I (December 2007); however, as of April 2009, the NHSP has yet to be published.

NIMS presents a core set of doctrine, concepts, principles, procedures, organizational processes, terminology, and standard requirements designed to enable effective, efficient, and collaborative incident management. It forms the basis for interoperability and compatibility to enable a diverse set of public and private organizations to conduct integrated emergency management and incident response operations. (March 2004; revised December 2008) (interim NPG, March 2005; final NPG and 3 capabilities-based preparedness tools, September 2007)

The NPG consists of a vision, capabilities, and priorities for national preparedness. It establishes three capabilities-based preparedness tools (Universal Task List, Target Capabilities List, and National Planning Scenarios) and the National Preparedness System cycle (plan; organize and staff; equip; train; and exercise, evaluate, and improve) to collate existing homeland security plans, strategies, and systems into an overarching framework. The revised and finalized NPG replaces the interim National Preparedness Goal released in March 2005.

The NRF is a guide to how the nation conducts all-hazards response, generally describing national response doctrine and the roles and responsibilities of officials involved in response efforts, including, among others, the Secretary of Homeland Security, FEMA Administrator, Principal Federal Official, and Federal Coordinating Officer. It is designed to align key roles and responsibilities, linking all levels of government, nongovernmental organizations, and the private sector, as well as capture specific authorities and best practices for managing incidents that range from the serious but purely local, to large-scale terrorist attacks or catastrophic natural disasters. (January 2008)

NRF ESF Annexes align categories of federal government response resources and capabilities and provide strategic objectives for their use under the NRF. They provide the structure for coordinating federal interagency support for a federal response to an incident and are mechanisms for grouping functions most frequently used to provide federal support to states, and federal-to-federal support. Each ESF Annex, such as Search and Rescue, identifies the federal agency coordinator and the primary and support agencies pertinent for the ESF, and when activated, the initial actions delineated in the ESF Annexes guide response activities. (January 2008)

NRF Support Annexes describe the roles and responsibilities of federal departments and agencies, nongovernmental organizations, and the private sector in coordinating and executing the functional processes and administrative requirements necessary for incident management that are common to all incidents. They identify the federal agency coordinator and the primary and support agencies pertinent for the support activity, such as financial management or private-sector coordination, and when activated, the initial actions delineated in the Support Annexes guide response activities. (January 2008)

NRF Partner Guides are to provide stakeholder-specific references describing key roles and actions for local, tribal, state, federal, private- sector, and nongovernmental response partners. They are to summarize core NRF concepts and be tailored specifically to leaders at different levels of government and from different types of organizations. As of April 2009, none of the four NRF Partner Guides have been published. (release schedule undetermined)

Completed (April 2006)

Completed (April 2006)

The various Joint Field Office (JFO) guides were written to support the December 2004 National Response Plan for the establishment of JFOs. The JFO is a temporary federal multi-agency coordination center established locally to facilitate coordinated field-level domestic incident management activities, with the JFO IISOP, JFO A&A, and JFO FOG designed to assist personnel assigned to response operations. In particular, the JFO IISOP and JFO A&A provide detailed guidance on JFO activations and operations, including defining the roles and responsibilities and concept of operations for the Principal Federal Official cell within the JFO. The JFO FOG is intended to be used as a quick reference job aid for JFO personnel.

Completed (June 2006)

According to FEMA officials, FEMA’s Disaster Operations Directorate is developing a JFO Organization & Functions Manual, reflecting the NRF, as an addition to existing JFO guidance. This new manual is to provide guidance for the establishment, operation, and demobilization, as well as the general organization and staffing, of JFOs (release schedule undetermined)

The type column identifies the categorization of policy and plans. Policies are categorized under the term “policy & doctrine” (including legislation, presidential directives, and other policies that define roles and responsibilities, as well as policies that define planning processes), per the national preparedness cycle presented in the National Preparedness Guidelines. Plans that define roles and responsibilities and that are to operationalize policies are categorized under the term “planning & resource allocation,” also per the national preparedness cycle presented in the National Preparedness Guidelines.

Table 5 provides brief descriptions and the status of the policies that define planning processes for developing the emergency plans depicted in figure 7. Of the four policies presented in the table, two have been completed and two are partially completed.

The range and relative relationships of the plans that define roles and responsibilities are illustrated in figure 8. Table 6, which follows figure 8, describes each of the plans presented in the graphic.

Table 6 provides brief descriptions and the status of the plans that define roles and responsibilities that are depicted in figure 8. Of the 72 plans presented in the table, 20 have been completed, 3 have been partially completed, and 49 are incomplete.

As noted in table 6, a schedule exists for the release of the myriad plans called for under HSPD 8 Annex 1 using the not yet published Integrated Planning System (IPS). Figure 9 presents the schedule for the release of specific plans under IPS.

Finally, figure 10 shows the combined universe of all the policies and plans in figures 7 and 8 in relation to each other. Figure 11, which follows, shows the status of development of each of the policies and plans.

Appendix III: Information Regarding Assessments Used to Develop the Comprehensive Assessment System

This appendix presents additional information regarding the Federal Emergency Management Agency’s (FEMA) progress and any remaining issues it faces in conducting a nationwide comprehensive assessment system. FEMA has identified various ongoing and historical assessment efforts that it plans to use to inform the development of the comprehensive assessment system. Additional information regarding these efforts is outlined below.

State Preparedness Reports. While state preparedness reports assess capabilities within states, FEMA could not use information in the 2007 reports to compare capability gaps between states, because states did not report information using common metrics to assess capabilities and data were not always available to consistently complete the report. FEMA’s state preparedness guidance explains that states are to “use relevant metrics . . . from the Target Capabilities List when describing current capabilities.” However, FEMA has not developed a framework for states to use in reporting their current capabilities against the target capabilities because FEMA is in the process of (1) developing quantifiable metrics for the target capabilities and (2) revising the reporting format for state preparedness reports in order to base them on the target capabilities. As a result, the 2007 reports do not report state capabilities in a measurable way, or with the level of detail necessary for a comparison across states and territories. In addition, the six states we visited used different techniques to summarize their capabilities. In one location, a state homeland security task force held discussion groups to determine what the capability needs are and what resources are needed. In another location, the state held a workshop attended by stakeholders from across the state to collect and obtain input on capability needs to complete the state preparedness report. Two states relied on information collected for their respective state homeland security strategies. A fifth state primarily used information it had gathered to prepare a grant funding reporting requirement, while officials at a sixth state collected information through site visits. FEMA headquarters officials explained that they intend to use the target capabilities as the framework for future state preparedness reports.

National Incident Management System Compliance Assessment Support Tool (NIMSCAST). NIMSCAST is a Web-based tool to assess states’ and territories’ compliance with the NIMS, which is a standardized process by which emergency responders are to conduct integrated incident response operations, rather than a method for assessing capabilities. Assessing compliance with National Incident Management System requirements is one of the requirements for the comprehensive assessment system. In February 2009, FEMA indicated that it will use NIMSCAST to continue collecting data on NIMS compliance in addition to collecting capability and state preparedness report data through a survey it will distribute in 2009. FEMA noted that this effort will consolidate reporting requirement s and fulfill Post-Katrina Act requirements for the comprehensive assessment system. In addition, FEMA noted that it will use d collected through NIMSCAST related to compliance with incident management processes and procedures to directly inform the of two target capabilities: On-Site Incident Management and Emergency Operations Center Management. However, it is unclear how FEMA will integrate this tool with other features of the comprehensive assessment system to assess the remaining 35 capabilities that will not be directly informed by NIMSCAST data related to compliance with incident manag procedures.

FEMA and localities better target and measure the results of using federal grant funds. As its organizing structure, the C2C uses th National Planning Scenarios, such as a hurricane, earthquake, improvised explosive device, or anthrax attack. Data for the C2C are t be based on self assessments of capabilities from state preparedne plans, estimates of baseline capability, and the estimated relative capability improvement expected from a requested level of grant investment. To be used effectively and enable comparisons across jurisdictions in evaluating grant proposals, the state and local data for assessing state and local capabilities must be in the common language of target capabilities and have metrics that are compatible with C2C. These metrics are being developed by FEMA’s National Preparedness Directorate. However, grantee use of C2C will not be mandatory, a nd thus its ultimate value is yet to be determined. In developing the initiative, grant officials are considering ways to collect data from stakeholders with a minimal burden and integrate analyses re from the C2C into existing programs, plans, and procedures.

National Preparedness System (NPS). The NPS was discontinuedta. because it was time consuming and did not produce meaningful da This Web-based management information system was designed to serve as an inventory tool to measure a jurisdiction’s ability to deliver elements of planning, organization, equipment, training, and ex Although it was developed in response to Homeland Security Presidential Directive 8’s preparedness requirements, in conjunction with the Target Capabilities List, and pilot tested in 10 states, the system was discontinued by the Department of Homeland Security because officials said it was too time consuming to use, according to FEMA officials. Because it was only piloted, FEMA officials explained that it did not generate meaningful preparedness information from the data collected. According to FEMA budget documentation required by the Office of Management and Budget for major information technology investments, FEMA spent nearly $15 million in total on the system for 2006, 2007, and 2008 before it was discontinued. ercises.

Pilot Capability Assessment (PCA). The PCA was labor intensive he and did not generate meaningful data. This assessment, based on t 37 target capabilities, was also intended to measure jurisdictions’ progress in achieving needed target capabilities. While it was developed in response to HSPD 8 preparedness requirements and in conjunction with the Target Capabilities List and pilot tested in six states, FEMA officials said it was too labor intensive. Because only piloted, FEMA did not generate meaningful preparedness information from the data collected, according to FEMA officials.

Capability Assessment for Readiness (CAR). The CAR lacked controls for validating the accuracy of self-reported assessment data. This assessment was proposed as a one-time nationwide assessmen performance in areas such as planning and hazard management to assess a national set of emergency management performance criteria for FEMA grant recipients. FEMA committed to preparing this assessment in hearings before the Senate Committee on Appropriations. The assessment was conducted in 1997 but concerns reported by the DHS Inspector General in March 2006 regarding self- reporting and the lack of controls for validating information reported by states limited the reliability and, therefore, the value of the data. The assessment was conducted once in 1997.

Appendix IV: GAO’s Description of the Six Characteristics of an Effective National Strategy

In February 2004, we identified six desirable characteristics of an effective national strategy that would enable its implementers to effectively shape policies, programs, priorities, resource allocations, and standards and that would enable federal departments and other stakeholders to achieve the identified results. We further determined in that report that national strategies with the six characteristics can provide policy makers and implementing agencies with a planning tool that can help ensure accountability and more effective results. To develop these six desirable characteristics of an effective national strategy, we reviewed several sources of information. First, we gathered statutory requirements pertaining to national strategies, as well as legislative and executive branch guidance. We also consulted the Government Performance and Results Act of 1993, general literature on strategic planning and performance, and guidance from the Office of Management and Budget on the President’s Management Agenda. In addition, among other things, we studied past reports and testimonies for findings and recommendations pertaining to the desirable elements of a national strategy. Furthermore, we consulted widely within GAO to obtain updated information on strategic planning, integration across and between the government and its partners, implementation, and other related subjects.

We developed these six desirable characteristics based on their underlying support in legislative or executive guidance and the frequency with which they were cited in other sources. We then grouped similar items together in a logical sequence, from conception to implementation. The following sections provide more detail on the six desirable characteristics.

Purpose, scope, and methodology: This characteristic addresses why the strategy was produced, the scope of its coverage, and the process by which it was developed. For example, a strategy should discuss the specific impetus that led to its being written (or updated), such as statutory requirements, executive mandates, or other events like the global war on terrorism. Furthermore, a strategy would enhance clarity by including definitions of key, relevant terms. In addition to describing what it is meant to do and the major functions, mission areas, or activities it covers, a national strategy would ideally address its methodology. For example, a strategy should discuss the principles or theories that guided its development, the organizations or offices that drafted the document, or working groups that were consulted in its development.

Problem definition and risk assessment: This characteristic addresses the particular national problems and threats at which the strategy is directed. Specifically, this means a detailed discussion or definition of the problems the strategy intends to address, their causes, and operating environment. In addition, this characteristic entails a risk assessment, including an analysis of the threats to and vulnerabilities of critical assets and operations. If the details of these analyses are classified or preliminary, an unclassified version of the strategy should at least include a broad description of the analyses and stress the importance of risk assessment to implementing parties. A discussion of the quality of data available regarding this characteristic, such as known constraints or deficiencies, would also be useful.

Goals, subordinate objectives, activities, and performance measures: This characteristic addresses what the national strategy strives to achieve and the steps needed to garner those results, as well as the priorities, milestones, and performance measures to gauge results. At the highest level, this could be a description of an ideal end state, followed by a logical hierarchy of major goals, subordinate objectives, and specific activities to achieve results. In addition, it would be helpful if the strategy discussed the importance of implementing parties’ efforts to establish priorities, milestones, and performance measures that help ensure accountability. Ideally, a national strategy would set clear desired results and priorities, specific milestones, and outcome-related performance measures while giving implementing parties flexibility to pursue and achieve those results within a reasonable time frame. If significant limitations on performance measures exist, other parts of the strategy should address plans to obtain better data or measurements, such as national standards or indicators of preparedness.

Resources, investments, and risk management: This characteristic addresses what the strategy will cost, the sources and types of resources and investments needed, and where those resources and investments should be targeted. Ideally, a strategy would also identify appropriate mechanisms to allocate resources. Furthermore, a national strategy should elaborate on the risk assessment mentioned earlier and give guidance to implementing parties to manage their resources and investments accordingly. It should also address the difficult, but critical, issues about who pays and how such efforts will be funded and sustained in the future. Furthermore, a strategy should include a discussion of the type of resources required, such as budgetary, human capital, information, information technology (IT), research and development (R&D), procurement of equipment, or contract services. A national strategy should also discuss linkages to other resource documents, such as federal agency budgets or human capital, IT, R&D, and acquisition strategies. Finally, a national strategy should also discuss in greater detail how risk management will aid implementing parties in prioritizing and allocating resources, including how this approach will create society-wide benefits and balance these with the cost to society. Related to this, a national strategy should discuss the economic principle of risk-adjusted return on resources.

Organizational roles, responsibilities, and coordination: This characteristic addresses what organizations will implement the strategy, their roles and responsibilities, and mechanisms for coordinating their efforts. It helps to answer the question about who is in charge during times of crisis and during all phases of national preparedness: prevention, vulnerability reduction, and response and recovery. This characteristic entails identifying the specific federal departments, agencies, or offices involved, as well as the roles and responsibilities of private sectors. A strategy would ideally clarify implementing organizations’ relationships in terms of leading, supporting, and partnering. In addition, a strategy should describe the organizations that will provide the overall framework for accountability and oversight. Furthermore, a strategy should also identify specific processes for coordination and collaboration between sectors and organizations—and address how any conflicts would be resolved.

Integration and implementation: This characteristic addresses both how a national strategy relates to other strategies’ goals, objectives, and activities (horizontal integration)—and to subordinate levels of government and other organizations and their plans to implement the strategy (vertical integration). Similarly, related strategies should highlight their common or shared goals, subordinate objectives, and activities. In addition, a national strategy should address its relationship with relevant documents from implementing organizations, such as the strategic plans, annual performance plans, or the annual performance reports the Government Performance and Results Act requires of federal agencies. A strategy should also discuss, as appropriate, various strategies and plans produced by the state, local, or private sectors. A strategy also should provide guidance such as the development of national standards to link together more effectively the roles, responsibilities, and capabilities of the implementing parties.

Appendix V: Comments from the Department of Homeland Security

Appendix VI: GAO Contact and Staff Acknowledgments

William O. Jenkins Jr., (202) 512-8757 or jenkinswo@gao.gov.

Staff Acknowledgements

In addition to the contact named above, Chris Keisling (Assistant Director), Neil Asaba (Analyst-in Charge), Joel Aldape, Avrum Ashery, Tina Cheng, Brian Chung, Christine Davis, Lara Kaskie, Ron La Due Lake, Brian Lipman, David Lysy, Jan Montgomery, and Robert Robinson made key contributions to this report.

Related GAO Products

Actions Taken to Implement the Post-Katrina Emergency Management Reform Act of 2006. GAO-09-59R. Washington, D.C.: November 21, 2008.

Voluntary Organizations: FEMA Should More Fully Assess Organization's Mass Care Capabilities and Update the Red Cross Role in Catastrophic Events. GAO-08-823. Washington, D.C.: September 18, 2008.

Emergency Preparedness: States Are Planning for Medical Surge, but Could Benefit from Shared Guidance for Allocating Scarce Medical Resources. GAO-08-668. Washington, D.C.: June 13, 2008.

Emergency Management: Observations on DHS's Preparedness for Catastrophic Disasters. GAO-08-868T. Washington, D.C.: June 11, 2008.

National Response Framework: FEMA Needs Policies and Procedures to Better Integrate Non-Federal Stakeholders in the Revision Process. GAO-08-768. Washington, D.C.: June 11, 2008.

Homeland Security: DHS Improved its Risk-Based Grant Programs' Allocation and Management Methods, But Measuring Programs' Impact on National Capabilities Remains a Challenge. GAO-08-488T. Washington, D.C.: March 11, 2008.

National Disaster Response: FEMA Should Take Action to Improve Capacity and Coordination between Government and Voluntary Sectors. GAO-08-369. Washington, D.C.: February 27, 2008.

Continuity of Operations: Selected Agencies Tested Various Capabilities during 2006 Governmentwide Exercise. GAO-08-185. Washington, D.C.: November 19, 2007.

Homeland Security: Observations on DHS and FEMA Efforts to Prepare for and Respond to Major and Catastrophic Disasters and Address Related Recommendations and Legislation. GAO-07-1142T. Washington, D.C.: July, 31, 2007.

Homeland Security: Preparing for and Responding to Disasters. GAO-07-395T. Washington, D.C.: March 9, 2007.

Catastrophic Disasters: Enhanced Leadership, Capabilities, and Accountability Controls Will Improve the Effectiveness of the Nation's Preparedness, Response, and Recovery System. GAO-06-618. Washington, D.C.: September 6, 2006.

Emergency Preparedness and Response: Some Issues and Challenges Associated with Major Emergency Incidents. GAO-06-467T. Washington, D.C.: February 23, 2006.

GAO's Preliminary Observations Regarding Preparedness and Response to Hurricanes Katrina and Rita. GAO-06-365R. Washington, D.C.: February 1, 2006.

Homeland Security: DHS' Efforts to Enhance First Responders' All- Hazards Capabilities Continue to Evolve. GAO-05-652. Washington, D.C.: July 11, 2005.

Results-Oriented Government: Improvements to DHS's Planning Process Would Enhance Usefulness and Accountability. GAO-05-300. Washington, D.C.: March 31, 2005.

Homeland Security: Agency Plans, Implementation, and Challenges Regarding the National Strategy for Homeland Security. GAO-05-33. Washington, D.C.: January 14, 2005.

Homeland Security: Process for Reporting Lessons Learned from Seaport Exercises Needs Further Attention. GAO-05-170. Washington, D.C.: January 14, 2005.

Homeland Security: Federal Leadership and Intergovernmental Cooperation Required to Achieve First Responder Interoperable Communications. GAO-04-740. Washington, D.C.: July 20, 2004.