National Preparedness: FEMA Has Made Progress, but Needs to Complete and Integrate Planning, Exercise, and Assessment Efforts
Published: 2009-04-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Hurricane Katrina was the most destructive disaster in our nation's history and it highlighted gaps in preparedness for a catastrophic disaster. The Federal Emergency Management Agency (FEMA), a component within the Department of Homeland Security (DHS), is the lead federal agency responsible for developing a national preparedness system. The system includes policies and plans as well as exercises and assessments of capabilities across many public and private entities. GAO was asked to assess the extent to which FEMA has (1) developed policies and plans that define roles and responsibilities; (2) implemented the National Exercise Program, a key tool for examining preparedness; (3) developed a national capabilities assessment; and (4) developed a strategic plan that integrates these elements of the preparedness system. GAO analyzed program documents, such as after-action reports, and visited six states located in disaster regions. While the results of these visits are not generalizable, they show how select states carry out their efforts.
section_title: What GAO Found
paragraphs: While most policies (41 of 50) that define roles and responsibilities have been completed, such as the National Response Framework, 68 percent (49 of 72) of the plans to implement these policies, including several for catastrophic incidents, are not yet complete. As a result, the roles and responsibilities of key officials involved in responding to a catastrophe have not been fully defined and, thus, cannot be tested in exercises. The lack of clarity in response roles and responsibilities among the diverse set of responders contributed to the disjointed response to Hurricane Katrina and highlighted the need for clear, integrated disaster preparedness and response policies and plans. Although best practices for program management call for a plan that includes key tasks and their target completion dates, FEMA does not have such a plan. With such a plan, FEMA would be better positioned to ensure that the policies and plans are completed and integrated with each other as intended as well as with other elements of the preparedness system. Since 2007, FEMA has taken actions to implement the National Exercise Program at the federal and state levels by developing, among other things, program guidance and systems to track corrective actions; however, FEMA faces challenges in ensuring that the exercises are carried out consistent with program guidance. For example, the Homeland Security Council (an interagency entity responsible for coordinating homeland security policy) and state participants did not systematically track whether corrective actions had been taken to address deficiencies identified by exercises as called for by program guidance. As a result, FEMA lacks reasonable assurance that entities have taken actions aimed at improving preparedness. FEMA has made progress in developing a system for assessing national preparedness capabilities by, among other things, establishing reporting guidance for state preparedness, but it faces challenges in completing the system and required reports to assess preparedness. While FEMA has developed a project management plan for the new system, the plan does not fully identify milestones and program risks for developing quantifiable metrics necessary for measuring preparedness capabilities. A more complete project plan that identifies milestones and program risks would provide FEMA with greater assurance that it can produce a system to assess capabilities and inform decisions related to improving national preparedness. FEMA's strategic plan for fiscal years 2008-2013 recognizes that each of its components need to develop its own strategic plans that integrate the elements of national preparedness. FEMA's National Preparedness Directorate has yet to develop its strategic plan, but instead plans to use a draft annual operating plan to guide its efforts. This plan does not include all elements of a strategic plan, such as how the directorate will integrate the various elements of the system over time to improve national preparedness. Having a strategic plan would provide FEMA with a roadmap for addressing the complex task of guiding and building a national preparedness system.

section_title: Letter
paragraphs: The attacks of 9/11 and Hurricane Katrina were, respectively, the most  destructive terrorist and natural disasters in our nation’s history and  highlighted gaps in the nation’s readiness to respond effectively to large- scale catastrophes.  Among the gaps were a lack of clarity in roles and  responsibilities of federal, state, and local officials; limitations in assisting  those with special needs, such as residents in nursing homes; and  logistical problems in providing food, water, and other goods to those in  areas affected by the disaster.  To strengthen the nation’s preparedness for  such incidents, in December 2003 the President issued a directive that  called on the Secretary of Homeland Secretary to enhance capabilities of  federal, state, and local entities.  In October 2006, the Post-Katrina  Emergency Management Reform Act (Post-Katrina Act) charged the  Federal Emergency Management Agency (FEMA)—a major component of  the Department of Homeland Security (DHS)—with responsibility for  leading the nation in developing a national preparedness system.
The size and complexity of the nation’s preparedness activities and the  number of organizations involved make developing a national  preparedness system a difficult task.  To lead national preparedness  efforts effectively, FEMA is to coordinate with a wide range of federal  departments and agencies, such as the Departments of Defense, Health  and Human Services, and Justice; 50 states, the District of Columbia, and 5  territories; city and county governments; a wide range of nonprofit  organizations; and private entities, such as debris removal contractors.   Each of these entities has many stakeholders with whom FEMA is to  coordinate and partner, such as governors and mayors, state and local  emergency managers, and first responders.
The national preparedness system, as shown below in figure 1, is a  continuous cycle of (1) establishing policy and doctrine, (2) planning and  allocating resources, (3) conducting training and exercises to gather  lessons learned, and (4) assessing and reporting on the training and  exercises to evaluate preparedness, including identifying any gaps in  capabilities.  The results of these assessments and reports are then used to  inform decision makers on what improvements are needed in policies and  plans and how to target finite resources to improve preparedness for  disasters.
Since 2005, we have issued several reports on the progress, challenges,  and weaknesses that exist in carrying out various preparedness programs,  including exercises and assessment efforts.  Given the critical role that  FEMA plays in homeland security and emergency response, you asked us  to review the progress that FEMA has made to develop key elements of  the national preparedness system.  As a result, this report examines  several key parts of the system in more detail:  policy and doctrine;  planning, exercises and lessons learned; and assessments and reporting.   Policies broadly define roles and responsibilities, while plans  operationalize policies by providing more specific information on roles  and responsibilities and associated tasks.  FEMA’s National Exercise  Program carries out exercises which are a key tool for testing and  evaluating preparedness.  Assessments include not only assessments of  individual exercises and other preparedness efforts, but of capabilities to  determine what gaps, if any, exist in capability levels.  FEMA’s ability to  establish the national preparedness system depends, in part, on the  effectiveness with which it integrates the elements of the system and  coordinates with stakeholders, including federal, state, and local agencies,  involved in national preparedness.
Specifically, this report addresses the extent to which FEMA has:    developed policies and plans to define roles and responsibilities and  planning processes for national preparedness;    taken actions since 2007 to implement the National Exercise Program  and to track corrective actions at the federal and state levels and what  challenges remain;    made progress in conducting a nationwide capabilities-based  assessment, including developing required preparedness reports, and  what issues, if any, it faces in completing the system; and     developed a strategic plan for implementing the national preparedness  system.
To address the questions above, we analyzed information and data on  FEMA’s policies and plans for preparedness, implementation of the  National Exercise Program, its approach for developing a comprehensive  system for assessing nationwide capabilities, and its strategy for  integrating elements of the preparedness system.  We analyzed key  legislation such as the Post-Katrina Act and the Implementing  Recommendations of the 9/11 Commission Act of 2007 (“9/11 Act”) as well  as presidential directives related to preparedness efforts.  We interviewed  FEMA officials responsible for preparedness programs to learn more  about the actions they had taken and planned to take related to  preparedness efforts.   Specifically,    To analyze the extent to which FEMA has developed policies and plans  to define roles and responsibilities and planning processes for national  preparedness, we reviewed the policies and plans that form the basis of  the preparedness system.  These policies and plans include, among  others, the National Response Framework (NRF) and National  Preparedness Guidelines, as well as the national integrated planning  system and preliminary versions of related guidance to develop and  integrate plans across federal, state, tribal, and local governments.8,  We summarized lessons learned from after-action reports for federal  exercises from 2005 through 2008 to determine whether the exercises  revealed unclear or conflicting roles and responsibilities between  federal departments and agencies, and whether additional policies or  plans were identified as needed corrective actions.  We also compared  FEMA’s approach for ensuring policies and plans are completed with  best practices for program management established by the Project  Management Institute.
Issued by DHS in January 2008, the NRF is the doctrine that guides how federal, state,  local, and tribal governments, along with nongovernmental and private sector entities, will  collectively respond to and recover from all hazards, including catastrophic disasters such  as Hurricane Katrina.  The NRF consists of a core base document that broadly describes  the roles and responsibilities of key officials.  In addition, roles and responsibilities for  response activities are further defined by 15 NRF Emergency Support Function (ESF)  Annexes, while 8 additional NRF Support Annexes describe the execution of common  incident management functional processes and administrative requirements.  The NRF also  includes 4 Partner Guides, which are to provide ready references describing key roles and  actions for local, tribal, state, federal, private-sector, and nongovernmental response  partners.  Finally, the NRF contains 7 NRF Incident Annexes and and 2 NRF Incident  Annex Supplements, which are to address aspects of how the United States responds to  broad incident types.
Titled “National Planning,” Homeland Security Presidential Directive 8 Annex 1 is intended  to enhance the preparedness of the United States by formally establishing, developing, and  maintaining a standard and comprehensive approach to national planning. It calls for,  among other things, a standardized federal planning process (the Integrated Planning  System); a family of strategic guidance statements, strategic plans, concepts of operations,  operations plans, and, as appropriate, tactical plans for responding to each National  Planning Scenario; and a system for integrating plans among all levels of government. preparedness.  We also interviewed Homeland Security Council  (HSC) staff and the Associate Counsel to the President on the rol responsibilities of the council in designing, executing, and following up  on corrective actions resulting from Principal Level Exercises because  these staff were responsible for summarizing lessons learned and  corrective actions for such exercises.  For information that would  provide a broader perspective on FEMA’s efforts, we examined several  FEMA databases, including the FEMA Secure Portal—the FEMA  repository of after-action reports; the National Exercise Schedule  (NEXS) system—a scheduling system for all exercises; and the  Corrective Action Program (CAP) System—which was designed for  tracking capability improvement plans entered by federal, state, and  local exercise participants.  We assessed the reliability of these  databases by checking them against documents, such as after-action  reports produced by states, and by interviewing staff responsible for  the data.  We concluded the data in the FEMA Secure Portal, NEXS  system, and CAP system were not reliable for use in this report.  We  discuss the reliability of these data later in this report.  Finally, we  compared FEMA’s policies and procedures for implementing the  National Exercise Program with criteria in GAO’s standards for internal  control in the federal government.
To determine the extent to which FEMA has made progress and what  issues, if any, remain in conducting a nationwide capabilities-based  assessment and developing required preparedness reports, we analyzed  information pertinent to FEMA’s assessment approach, such as State  Preparedness Reports and the Federal Preparedness Report.  We also  examined FEMA’s plans for implementing the comprehensive  assessment system.  We interviewed FEMA staff responsible for  developing FEMA’s assessment system and performance objectives for  measuring capabilities.  Finally, we reviewed our prior reports on  FEMA’s preparedness programs.
To determine the extent to which FEMA has developed a strategic plan  that integrates the national preparedness system, we analyzed key  FEMA documents including the agencywide Strategic Plan for Fiscal  Years 2008 through 2013, the Grant Programs Directorate Strategic  Plan, and the draft annual National Preparedness Directorate Operating  Plan.  We interviewed FEMA National Preparedness Directorate staff  on strategic planning and policy and procedures for the national  preparedness system.  We also compared FEMA’s current approach for  developing a national preparedness system with the desirable  characteristics of effective national strategies we identified in February  2004.
To supplement our analyses of FEMA’s preparedness activities, we visited  six states—California, Georgia, Illinois, New York, Texas, and Washington.   While we cannot generalize our work from these visits to all states, we  chose these locations for their geographic diversity and to provide  examples of the way in which states carry out their exercise and  preparedness programs.  In selecting these states, we considered factors  such as states’ participation in national level exercises; states located in  hurricane-prone regions; and states with varying percentages of homeland  security grant funding planned to support exercises. At each location, we  interviewed staff in FEMA’s regional offices responsible for implementing  regional requirements of the National Exercise Program and regional  preparedness activities.  We also interviewed officials in each state on  their progress and challenges in carrying out preparedness activities,  including exercises and assessments of capability.  In addition, we  evaluated after-action reports produced by each of the six states.
We conducted this performance audit from January 2008 through April  2009 in accordance with generally accepted government auditing  standards.  Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives.  We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.  Appendix I contains more  details on our scope and methodology.
subsections: section_title: Background
paragraphs: In 2005, Hurricane Katrina dramatically illustrated the adverse  consequences that can occur when the nation is unprepared to respond  effectively to a catastrophic disaster.  Emergency preparedness  strengthens the nation’s ability to prevent, protect, respond to, and recover  from a natural disaster, terrorist attack, or other man-made disaster.  It has  received widespread attention and support from Congress, the President,  and the Secretary of Homeland Security as manifested by legislation,  presidential directives, the development of DHS policy documents, and  grants to state and local governments.  The lessons learned from the  terrorist attacks of 9/11 and Hurricane Katrina focused attention on the  need for preparedness programs that could (1) guide decisions on how to  improve policies and plans that define roles and responsibilities across the  broad spectrum of governmental and nongovernmental organizations  involved in prevention, protection, response, and recovery activities and  (2) help managers prioritize the use of finite resources to narrow gaps in  needed capabilities.
FEMA—a component in DHS—is the federal agency responsible for  leading the nation’s preparedness activities.  In December 2003, the  President issued guidance that called on the Secretary of Homeland  Security to carry out and coordinate preparedness activities with public,  private, and nonprofit organizations involved in such activities.  In the  wake of the problems that marked the response to Hurricane Katrina in  2005, Congress passed the Post-Katrina Act in October 2006. The act  strengthened FEMA’s role within DHS and defined FEMA’s primary  mission as: “to reduce the loss of life and property and protect the Nation  from all hazards, including natural disasters, acts of terrorism, and other  man-made disasters, by leading and supporting the nation in a risk-based,  comprehensive emergency management system of preparedness,  protection, response, recovery, and mitigation.”  The act required FEMA  to establish a national preparedness system for ensuring that the nation  has the ability to deal with all hazards, including those incidents with  catastrophic consequences.  Among other things, the act directs FEMA to  provide funding, training, exercises, technical assistance, planning, and  other assistance to build tribal, local, state, regional, and national  capabilities (including communications capabilities) necessary to respond  to any type of disaster.  It also requires FEMA to develop and coordinate  the implementation of a risk-based all-hazards strategy for preparedness  that builds those common capabilities necessary to any type of disaster,  while also building the unique capabilities necessary to respond to specific  types of incidents that pose the greatest risk to the United States.  The  act includes a number of other specific requirements including the  development of quantifiable performance measurements to support each  component of the national preparedness system, such as capabilities  assessment, training, and exercises.  The system provides a basis for  improvements in policies, plans, and capabilities that aim to save lives and  protect and preserve property.
subsections: section_title: National Preparedness Aims to Clarify Roles and Responsibilities and Close Gaps in Disaster Response Capability
paragraphs: DHS has defined national preparedness as a continuous cycle that involves  four main elements: (1) policy and doctrine, (2) planning and resource  allocation, (3) training and exercises, and (4) an assessment of capabilities  and reporting.  The following is a brief description of each element of the  system.
Policy:  This element involves ongoing management and maintenance of  national policy and doctrine for operations and preparedness.  Disaster  response is primarily handled by local or tribal governments with the state  and federal governments and private and nonprofit sectors playing  supporting and ad hoc roles, respectively, as needed or as requested.   One of the lessons learned from Hurricane Katrina is that a lack of clarity  regarding roles and responsibilities across these levels of government and  sectors can result in a less coordinated national response and delay the  nation’s ability to provide life-saving support when needed.  Broadly  speaking, FEMA’s role, in cooperation with other federal and nonfederal  entities, is to define the roles and responsibilities for all response  stakeholders so that each understands how it supports the broader  national response.  This approach calls for a national response based on  partnerships at and across all levels.
Planning and resource allocation:  This element involves application of  common planning processes and tools by government officials, working  with the private sector, nongovernmental organizations, and individual  citizens to identify requirements, allocate resources, and build and  maintain coordinated capabilities that are prioritized based upon risk.   Among other things, this element involves developing planning processes  so that roles and responsibilities of stakeholders are clearly defined for  specific homeland security scenarios, such as a hurricane or a terrorist  attack involving nuclear or radiological weapons.
Training and exercises:  Exercises provide opportunities to test plans  and improve proficiency in a risk-free environment.  Exercises assess the  adequacy of capabilities as well as the clarity of established roles and  responsibilities.  Short of performance in actual operations, exercise  activities provide the best means to evaluate returns on homeland security  investments.  The Post-Katrina Act requires FEMA to carry out a National  Training Program and National Exercise Program.  On January 26, 2007,  the National Security Council and the HSC approved the establishment of  a new iteration of the National Exercise Program to conduct exercises to  help senior federal government officials prepare for catastrophic crises  ranging from terrorism to natural disasters.  Well-designed and executed  exercises can improve interagency coordination and communications,  highlight capability gaps, and identify opportunities for improvement.   Tracking corrective actions resulting from exercises is a key step in the  process.
Assessing capabilities and reporting:  According to the Post-Katrina  Act, FEMA is required to develop a comprehensive assessment system to  assess the nation’s prevention capabilities and overall preparedness.  A  key part of the system involves the development of quantifiable standards  and metrics—called target capabilties—that can be used to assess existing  capability levels compared with target capability levels.  The act requires  FEMA to include the results of its comprehensive assessments in an  annual Federal Preparedness Report to Congress.  To assist in this effort,  FEMA is to receive annual State Preparedness Reports from all 50 states,  the District of Columbia, and 5 territories that receive DHS preparedness  assistance.
FEMA’s National Preparedness Directorate has primary responsibility for  carrying out the key elements of the national preparedness system, in  coordination with other federal, state, local, tribal, nonprofit, and private- sector organizations.  The directorate includes the National Integration  Center and the Office of Preparedness Policy, Planning, and Analysis  (PPPA).  The National Preparedness Directorate and FEMA’s Disaster  Operations Directorate share responsibility for ensuring plans that  describe roles and responsibilities are developed.  FEMA’s National  Exercise Division—a division of the National Integration Center—leads  exercise activities.  Finally, PPPA is responsible for assessing capabilities.   See figure 2 for an organizational chart of select FEMA components  involved in preparedness programs.
section_title: Developing and Integrating the Elements of the National Preparedness System Is a Challenging Responsibility
paragraphs: FEMA faces two main challenges in developing and integrating the  elements of the national preparedness system.
For example, see GAO, National Response Framework:  FEMA Needs Policies and  Procedures to Better Integrate Non-Federal Stakeholders in the Revision Process,  GAO-08-768 (Washington, D.C.: June 11, 2008) and Voluntary Organizations:  FEMA  Should More Fully Assess Organization’s Mass Care Capabilities and Update the Red  Cross Role in Catastrophic Events, GAO-08-823 (Washington, D.C.: Sept. 18, 2008).
See Homeland Security Advisory Council, Top Ten Challenges Facing The Next Secretary  of Homeland Security (Washington, D.C.: Sept. 11, 2008).  Among other things, this report  determined that the work of strengthening disaster response capabilities is incomplete, in  part, because DHS will need to ensure involvement of homeland security partners in  building a bottom-up approach of organization and response as it establishes national  planning efforts.
The National Council on Disability is an independent federal agency and is composed of  15 members appointed by the President. It provides advice to the President, Congress, and  executive branch agencies to promote policies, programs, practices, and procedures that  guarantee equal opportunity for all individuals with disabilities, regardless of the nature or  severity of the disability and to empower individuals with disabilities to achieve economic  self-sufficiency, independent living, and inclusion and integration into all aspects of  society. involves these preparedness stakeholders in plans that define roles a responsibilities, exercises, and assessments of capabilities.
6 U.S.C. § 315.
March 2008 half of the staff positions in the PPPA office were vacant  and these positions were not filled until November 2008, according to  FEMA officials.  In addition, FEMA did not hire a permanent director  for the PPPA office until October 2008.  Reorganization of the National  Exercise Program also brought about staffing changes, according to  FEMA officials.  Among other things, in the fall of 2008 FEMA hired a  new director for the National Exercise Division—a position that had  been vacant for several months, according to FEMA officials.  Fig. 3  illustrates the organizational changes related to the National Exercise  Program.
section_title: Most Policies That Define Roles and Responsibilities and Planning Processes Have Been Completed, but Work Remains to Complete Emergency Plans
paragraphs: Defining roles and responsibilities is a key step in developing the national  preparedness system.  While most key policies that define roles and  responsibilities have been completed, 68 percent (49 of 72) of plans that  operationalize such policies have not been completed.  Lessons learned  from Hurricane Katrina and emergency response exercises demonstrated  the need for the development of complete policies and plans to address  potential problems with stakeholders not understanding their roles and  responsibilities in response to a catastrophic disaster.  Although best  practices for program management state that a program management plan  is an essential tool for implementing a program, FEMA, in coordination  with DHS and other federal entities, has not yet fully developed such a  plan to help ensure the development and integration of policies and plans  that define roles and responsibilities and planning processes for  emergency response.
subsections: section_title: Defining Roles and Responsibilities Is Key to Developing the Preparedness System
paragraphs: Legislation and presidential directives call for the development of policies  and plans that define roles and responsibilities, which is key to FEMA’s  ability to develop the preparedness system.  For example, the Homeland  Security Act of 2002, as amended by the Post-Katrina Act, requires FEMA  to consolidate existing federal government emergency response plans into  a single, coordinated National Response Plan, now known as the National  Response Framework. To develop the preparedness system, FEMA is to  partner and coordinate with key stakeholders, including other DHS  components as well as other federal, state, and local entities.  For  example, the Post-Katrina Act requires the FEMA Administrator, under the  leadership of the Secretary of Homeland Security, to coordinate with other  agencies and offices in DHS to take full advantage of their range of  available resources.  In addition, the Post-Katrina Act requires FEMA to  coordinate with other federal departments and agencies and the National  Advisory Council to develop and refine key national policy documents that  broadly define roles and responsibilities, including the National Incident  Management System (NIMS) and the NRF.
Plans developed using established planning processes operationalize  policy documents, such as the NRF, by providing additional details on the  roles and responsibilities for each individual and organization that may be  involved in responding to high-risk or catastrophic incidents.  For  example, during the planning process, FEMA may need to coordinate with  other federal departments and agencies—such as the Department of  Defense (DOD), Federal Bureau of Investigation, and other components in  DHS—to define roles and responsibilities for responding to a chemical  attack.  Per the Post-Katrina Act, each federal agency with responsibilities  under the NRF is responsible for developing its own operational plans.
The act further requires the President to certify to selected committees of  Congress on an annual basis that each federal agency has complied with  statutory requirements in the development of its operational plan,  including a requirement that the plan “be coordinated under a unified  system with a common terminology, approach, and framework.”  Although the Post-Katrina Act does not charge FEMA with developing or  certifying the federal agency plans, FEMA is statutorily responsible for the  basic architecture of the national preparedness system, in coordination  with other federal departments and agencies, among others.  This  principle of integration and coordination is also embodied in Homeland  Security Presidential Directive 8 Annex 1 (HSPD 8 Annex 1), which tasked  DHS with developing a national planning system to develop and integrate  plans that define preparedness roles and responsibilities, both horizontally  across federal departments and agencies (i.e., integration of plans that  have been developed by more than one federal department or agency) and  vertically with state and local emergency response plans (i.e., integration  of plans that have been developed by more than one level of government).   Thus, the federal preparedness framework depends on DHS’s—and, in  particular, FEMA’s—ability to coordinate with its federal department and  agency partners to ensure that their plans are integrated in a way that  avoids duplication of effort and confusion during an interagency response.
section_title: Most Key Policies That Define Roles and Responsibilities and Planning Processes Have Been Completed
paragraphs: DHS, FEMA, and other federal entities with a role in national preparedness  have completed most of the key policies that broadly define roles and  responsibilities and planning processes for developing more detailed  emergency plans.  Among the 50 policies that define roles and  responsibilities or planning processes, 42 have been completed, 2 have  been partially completed, and the remaining 6 are incomplete.
A more detailed breakdown of the 50 policies shows that 46 define roles  and responsibilities and 4 define planning processes for developing  emergency plans.  Forty of the 46 policies that define roles and  responsibilities have been completed.  Among the policies that have been  completed, for example, DHS issued the revised NIMS in December 2008  to further clarify roles and responsibilities when multiagency,  intergovernmental entities are involved in a response and to address, in  part, the confusion about roles and responsibilities that resulted in a poor  response to Hurricane Katrina.  Key components of the NRF have also  been completed, including the base NRF document, 15 Emergency  Support Function Annexes, and 8 Support Annexes.  However, other  components of the NRF have not been completed, such as the 4 Partner  Guides that are to provide abbreviated descriptions of the key roles and  responsibilities of specific federal, state, local, and private sector and  nongovernmental stakeholders under the NRF.  Also uncompleted is the  development of new Joint Field Office (JFO) guidance under the NRF  which, according to FEMA officials, is to provide functional guidance for  the organization and staffing of JFOs, as well as their establishment,  operation, and demobilization.  In addition, DHS has not completed the  National Homeland Security Plan, which is to serve as an overarching  strategic plan to guide national efforts to execute the National Strategy for  Homeland Security.  For details on, and the status of, all 46 policies that  are to define roles and responsibilities, see table 4 in appendix II.
In addition to the 46 policies that define roles and responsibilities, 4 other  policies are to define planning processes for developing emergency  plans.  Two of these 4 policies have been completed.  According to officials, FEMA issued planning guidance for its operational planners in  March 2009 and will update this guidance each fiscal year.  The other 2  policies that define planning processes have been partially completed—1  has been issued as an interim policy (Comprehensive Preparedness Guide  301) while the other has been drafted (Integrated Planning System) and is  being used by federal interagency incident management planners although  it has not been publicly released.  Additional details describing these 4  policies and their status are included in table 5 in appendix II.
section_title: Although Action Has Been Taken to Develop and Complete Some Plans That Define Roles and Responsibilities, 68 Percent of the Plans Have Not Been Completed
paragraphs: While DHS, FEMA, and other federal entities with a role in national  preparedness have taken action to develop and complete some plans that  detail and operationalize roles and responsibilities for federal and  nonfederal entities, these entities have not completed 68 percent of the  plans required by existing legislation, presidential directives, and policy  documents as of April 2009.  Specifically, of the 72 plans we identified, 20  have been completed (28 percent), 3 have been partially completed (that  is, an interim or draft plan has been produced—4 percent), and 49 (68  percent) have not been completed.  Detailed plans supplement and  operationalize key policy documents.  Among the plans that have been  completed, FEMA published the Pre-Scripted Mission Assignment Catalog  in 2008, which defines roles and responsibilities for 236 mission  assignment activities to be performed by federal government entities, at  the direction of FEMA, to aid state and local jurisdictions during a  response to a major disaster or an emergency.  See table 6 in appendix II  for additional details on the other 19 plans that have been completed.
One of the three plans that FEMA has partially completed is the Federal  Contingency Plan—New Madrid Seismic Zone Catastrophic Earthquake.   This plan addresses major issues the federal government expects to  encounter if a catastrophic earthquake occurs in the New Madrid Seismic  Zone with no warning.  FEMA published this plan in an interim form in  June 2008 and intends to finalize it by May 2010.  While FEMA has engaged  in significant planning efforts regarding threats that are specific to certain  regions, such as hurricanes and earthquakes, through its Catastrophic  Disaster Planning Initiative, those planning efforts are ongoing and have  not been concluded.  See table 6 in appendix II for additional details on  the other two plans that have been partially completed.
Among the 49 plans that have not been completed are the NRF incident  annexes for terrorism and cyber incidents as well as the NRF incident  annex supplements for catastrophic disasters and mass evacuations.  The  NRF incident annexes and incident annex supplements are to address the  roles and responsibilities and unique aspects of how the United States  responds to broad incident types.  In addition, operational plans for  responding to the consolidated national planning scenarios, as called for in  HSPD 8 Annex 1, remain outstanding.  For additional details and the status  of each of these plans, as well as other plans that are to define roles and  responsibilities, see table 6 in appendix II.
Developing plans to operationalize policies that define roles and  responsibilities is one key to an effective response.  According to DHS,  effective response hinges upon well-trained leaders and responders who  have invested in response preparedness, developed engaged partnerships,  and are able to achieve shared objectives.  Until outstanding policies and  plans, especially those that are new, are completed, FEMA, in  coordination with DHS and other federal departments and agencies,  cannot provide associated training on such policies and plans, and,  relatedly, cannot validate the new policies, plans, and training though  exercises—the next step in the national preparedness system cycle.
According to FEMA, certain plans that have yet to be completed are  refinements of existing plans, and training has been provided on the  existing plan.  However, such existing training will need to be adapted and  modified to enable response stakeholders to be trained on the revised  plans.  Incomplete policies and plans, especially those that are new, and  the resulting lack of associated training and validation through exercises,  increase the risk that response to an incident may be disjointed, delayed,  or ineffective because stakeholders may not understand their roles and  responsibilities.
The issue of completing emergency policies and plans that define roles  and responsibilities is not new.  Prior to Hurricane Katrina, based on a  summary of lessons learned from exercises conducted in fiscal year 2005,  DHS determined that plans for specific incidents were potentially needed  to clarify how the National Response Plan (now the NRF) would be  implemented under several types of domestic scenarios, such as the threat  of an improvised nuclear device detonation, large-scale biological events,  and suicide bombings.  In February 2006, the White House issued its report  on lessons learned from the response to Hurricane Katrina and concluded  that two of four critical deficiencies in the response involved policies and  plans, or the lack thereof, that were to detail roles and responsibilities.   Among other things, the report noted that federal departments and  agencies were required to develop supporting operational plans and  standard operating procedures for national response activities, but in  almost all cases, these required plans and procedures were either  nonexistent or still under development.  In addition, the report stated that  additional structural deficiencies in the national preparedness system  included weak regional response planning and coordination structures.   Further, in our September 2006 report on Hurricane Katrina, we  recommended that FEMA develop detailed and robust operational  implementation plans for the National Response Plan (now NRF) and its  Catastrophic Incident Annex and Supplement in preparation for and  response to future catastrophic disasters.
In addition, in October 2006, the Post-Katrina Act required FEMA to  develop prescripted mission assignments and each federal agency with  NRF responsibilities to develop operational plans and corresponding  capabilities in support of the NRF to ensure a coordinated federal  response.  In December 2007, the President signed HSPD 8 Annex 1,  which called for the development of an integrated set of scenario-based  response plans, including federal agency operational plans that are also  needed to satisfy the Post-Katrina Act.  More recently, in September 2008,  the Homeland Security Advisory Council (HSAC) for DHS identified the  work of strengthening the nation’s disaster response capabilities, which  includes developing and publishing outstanding emergency response  policies and plans that define roles and responsibilities for national  preparedness, as 1 of the top 10 DHS challenges for the incoming  presidential administration.
section_title: Despite Exercises Demonstrating the Need for the Development of Complete Policies and Plans, FEMA Has Not Fully Developed a Program Management Plan for Their Development and Integration
subsections: section_title: Recent Exercises Have Demonstrated the Importance of the Development of Complete Policies and Plans to Define Roles and Responsibilities
paragraphs: In addition to the lessons learned from Hurricane Katrina, recent exercises  have demonstrated the ongoing need for the development of complete  policies and plans that define roles and responsibilities for national  preparedness.   For example, we reviewed 16 after-action reports for  national or principal level or equivalent exercises that were conducted  from 2005 to 2008, and all of the reports (16 of 16) called for further  clarification of roles and responsibilities across federal departments and  agencies or between federal and nonfederal organizations, as shown in the  following examples.
As a result of a February 2007 exercise that tested a joint response by  federal agencies to an attack using improvised explosive devices, HSC  determined that the federal government needed to further refine the  delineation of roles and responsibilities for the Departments of  Defense, Justice, and Homeland Security.  The same exercise also  identified the need for the federal government to increase outreach to  governors in order to achieve a common understanding of federal and  state governments’ respective roles and responsibilities during such an  incident.  Finally, this exercise demonstrated that policies and plans  were needed to clearly articulate, prior to an incident occurring, the  circumstances in which the use of military assets and resources for  civilian response is appropriate.  Not addressing these recommended  corrective actions, among others, may result in less efficient and  effective responses.
In a September 2007 exercise, the HSC identified the need for senior  federal officials to be rapidly informed of the possible courses of action  available to them in responding to a major disaster or an emergency.   To do so, HSC asserted that the senior federal officials’ respective  organizations should have in place detailed plans to inform and execute  the senior federal officials’ decisions, as such plans would serve to  better integrate interagency response activities.  HSC made this  recommendation in September 2007, and it was acted on by the  President through the establishment of HSPD 8 Annex 1 in December  2007, more than a year after the Post-Katrina Act called for the  development of federal agency operational plans.
section_title: FEMA Has Not Established a Program Management Plan for the Development and Integration of Outstanding Emergency Response Policies and Plans
paragraphs: FEMA has not established a program management plan, in coordination  with DHS and other federal entities, to ensure the development and  integration of outstanding policies and plans that are to define roles and  responsibilities and planning processes.  The Post-Katrina Act makes the  FEMA Administrator, in coordination with other entities, responsible for  the development of the national preparedness system.  According to the  National Preparedness Guidelines, the national preparedness system, in  part, consists of the policies and plans that define roles and  responsibilities and planning processes for developing emergency plans.   Although the Post-Katrina Act requires federal agencies to develop their  own operational plans, those plans are to be “coordinated under a unified  system with a common terminology, approach, and framework.” This  coordination and unification is central to FEMA’s mission as the lead  agency in charge of national preparedness, and it requires that the policies  and plans that have been called for are developed and integrated so that  emergency response roles and responsibilities and planning processes are  fully defined and implemented.
Best practices for program management, established by the Project  Management Institute in The Standard for Program Management, state  that managing a program includes, among other things, (1) establishing  clear and achievable objectives; (2) balancing the competing demands for  quality, scope, time, and cost; and, (3) adapting the specifications, plans,  and approach to the different concerns and expectations of the various  stakeholders involved in the program’s projects.  A key step in managing  a program involves developing a program management plan, which is an  approved document that defines how a program will be executed,  monitored, and controlled.  The program management plan defines the  tactical means by which the program will be carried out.  According to The  Standard for Program Management, a program management plan should,  among other things:  identify the specific schedule of activities that need to be performed to  complete and identify dependencies among policy and planning  development activities;  identify the types and quantities of resources required to perform, and  amount of time needed to complete, all policy and planning  development activities;     analyze activity sequences, durations, resource requirements, and  schedule constraints to create and update the policy and planning  project schedules; and     control for changes to the project schedules precipitated by outside  forces.
Because FEMA has not established, in coordination with DHS and other  federal entities, such a plan to ensure the development and integration of  outstanding policies and plans, it is unclear when the full set of policies  and plans will be completed, and FEMA cannot determine whether it or  other entities with policy and plan development responsibilities, such as  DHS, are on schedule.  Further, because FEMA cannot determine whether  other entities with policy and plan development responsibilities are on  schedule, it cannot determine when and how it will integrate into the  national preparedness cycle the range of policies and plans required by  legislation, presidential directives, and other policy.  Based in part on our  work, in February 2009 FEMA officials acknowledged that a program  management plan should be established.  Without active utilization of a  program management plan, FEMA, in coordination with DHS and other  federal entities, may experience unforeseen delays in completing its  efforts to develop and integrate these policies and plans that define roles  and responsibilities and planning processes.  While FEMA, in coordination  with DHS and other federal entities, may experience unanticipated or  uncontrollable delays in developing outstanding policies and plans, it  would be better positioned to identify the effect of those delays and assess  measures to mitigate them with a program management plan in place and  utilized.
section_title: FEMA Has Taken Actions to Implement the National Exercise Program, but Faces Challenges in Meeting Statutory and Program Requirements
paragraphs: FEMA has developed guidance to implement the National Exercise  Program; however, it faces challenges in meeting statutory and program  requirements in conducting the program.  These challenges have arisen  because FEMA lacks procedures that detail how it will work with federal  entities and monitor states to ensure these entities carry out program  requirements.  In addition, FEMA faces challenges in measuring the  effectiveness of the program because the databases it uses to measure  program performance are incomplete.
subsections: section_title: FEMA Has Developed Guidance and Other Tools to Assist in Designing and Implementing the National Exercise Program
paragraphs: Exercises are a key element of the national preparedness system.   The  purpose of the National Exercise Program is to test and improve the  nation’s ability to prevent, prepare for, and respond to events such as  terrorist attacks and natural and man-made disasters.   To meet this  purpose, exercises should test existing capabilities against desired, or  target capabilities as well as verify and validate policies and plans that  define roles and responsibilities.
Developing and implementing the National Exercise Program is a difficult  task because the magnitude of the effort involves coordinating with and  relying on the cooperation of other DHS components such as the Coast  Guard, numerous federal entities such as the Homeland Security Council  (HSC)—which is responsible for coordinating federal interagency  homeland security policy—and state governments, among others.  This  coordination is especially critical at the federal level because FEMA lacks  the authority to compel federal agencies to comply with program  requirements.  At the state level, FEMA is able to use its grant programs to  ensure that states follow program guidelines.  Since 2007, FEMA has taken  a number of actions to implement the National Exercise Program.  Several  of these actions are summarized in table 1.
FEMA has also identified four tiers of exercises that comprise the National  Exercise Program (see fig. 4).  Among the exercises that involve federal  interagency coordination are Tier I National Level Exercises, which are  operations-based exercises that evaluate existing national plans and  policies, in concert with federal and nonfederal entities; and Principal  Level Exercises, which are discussion-based exercises among senior  federal officials that examine emerging issues.
Officials at all six states we visited cited actions taken by FEMA to  implement the National Exercise Program as positive contributions to  their exercise efforts.  For example, officials in five of the six states  indicated that HSEEP guidance was beneficial because it establishes  consistency in how exercises are designed and conducted.  State officials  who were involved in a National Level Exercise indicated that it not only  allowed them to test and validate local emergency response plans, but also  gave them the opportunity to meet federal stakeholders as well as first  responders from neighboring counties and cities, thus enabling them to  establish working relationships before an event occurs.  Similarly, exercise  planners in New York noted that a benefit of FEMA’s regional training and  exercise plan workshop was the positive relations that were developed  with FEMA officials.
section_title: States Have Also Taken Actions to Implement Exercise Programs
paragraphs: In addition to actions by FEMA, the six states we visited have also taken  actions to implement exercise programs (FEMA considers exercises  conducted by states to be Tier IV exercises.) For example, exercise  planners in Washington conducted a half-day senior leadership workshop  for federal, state, and local officials in May 2008 that identified a number  of issues related, among other things, to intergovernmental  communications during an incident.  Based on the workshop, officials in  Washington identified corrective actions that could be taken. For example,  they identified the need to increase postdisaster communications to help  speed recovery efforts and said they were taking corrective actions to  address this issue.  In California, state exercise planners involved planners  from other departments or other states in their design and planning  workshops as a way to share information on exercise design and conduct,  thereby increasing the interagency participation in their exercise program  efforts.  According to the workshop sponsors, participants benefited from  the multidisciplinary participation in the conference.  In Illinois, the state  used real-world events, such as sporting events, to test or “exercise”  response by local law enforcement and first responders.  State exercise  officials said these efforts were an effective supplement to their regular  exercise program.
section_title: FEMA Faces Challenges in Meeting Statutory and Program Requirements in Conducting and Measuring the Effectiveness of the National Exercise Program
paragraphs: While FEMA has taken actions to implement the National Exercise  Program, it faces challenges in meeting statutory and program  requirements in conducting and measuring the effectiveness of the  program.  First, FEMA does not have procedures in place to detail how it  is to (1) work with other federal entities and (2) monitor states to help  ensure that these entities promptly prepare after-action reports and track  and implement corrective actions for federal- and state-level exercises.   Second, the National Exercise Program’s ability to simulate a catastrophic  event to strain, or “stress”, the preparedness system is limited.  Third,  FEMA lacks data to measure the effectiveness and progress of the  National Exercise Program.
subsections: section_title: FEMA Does Not Have Procedures to Help Ensure that After-Action Reports from National and Principal Level Exercises Are Promptly Prepared and That Corrective Actions Are Tracked and Implemented
paragraphs: FEMA must design the National Exercise Program to provide, among  other things, for the prompt development of after-action reports and plans  for quickly incorporating lessons learned into future operations.  In  addition, the Post-Katrina Act requires FEMA to establish a program to  conduct remedial action tracking and long-term trend analyses.   According to the implementation plan for the National Exercise Program,  after-action reports for Tier I exercises (National Level Exercises or  Principal Level Exercises) must be issued within 180 days, or 6 months,  after the completion of an exercise, and the release of an after-action  report should not be delayed to reach consensus on all issues identified  during the exercise.  FEMA executes Tier I exercises in coordination with  HSC and others.  Although the Post-Katrina Act does not give FEMA the  authority to compel other federal entities to comply with the objectives of  the National Exercise Program, it places responsibility for implementing  the National Exercise Program on FEMA, in coordination with other  appropriate federal agencies and other entities.  Therefore, it is incumbent  on FEMA to coordinate with HSC and other federal entities to better  ensure that FEMA obtains the information it needs to meet its statutory  responsibility to track corrective actions.  However, FEMA has not  ensured that after-action reports for Tier 1 exercises are issued in a  prompt manner, nor has it tracked and documented implementation of  corrective actions for such exercises.  These challenges occurred, in part,  because FEMA has not established procedures that detail how the agency  will work with other federal entities to ensure National Exercise Program  requirements are met.
For example, FEMA conducted a Tier I National Level Exercise—Top  Officials 4 (TOPOFF 4)—in October 2007, but as of February 2009, or more  than 15 months later, FEMA had not yet issued the after-action report or  tracked and implemented corrective actions.  When an after-action report  on an exercise is delayed and not provided to stakeholders, the “lessons  learned” from the exercise diminish in importance, limiting stakeholders’  ability to make improvements in preparedness.  In February 2009, FEMA  officials stated that the draft after-action report for TOPOFF 4 had been  written, and various departments and agencies have approved the report,  but it was not approved by DHS prior to the change in administration  related to the 56th Presidential Inauguration.  According to FEMA  officials, a complicating factor in releasing the report is the political  sensitivity of information, as those who write after-action reports may face  internal pressure not to identify weaknesses in an entity’s emergency  preparedness.  As a result of these delays, stakeholders may not be able to  promptly make improvements in preparedness.
FEMA has also had limited success in ensuring that program requirements  for the National Exercise Program have been followed for Principal Level  Exercises.  HSEEP guidance and the implementation plan state that the  results of exercises should be documented and corrective action  responsibilities assigned and tracked.  Specifically, HSEEP guidance calls  for organizations to (1) complete an after-action report with an  improvement plan and (2) track corrective actions to ensure that they are  implemented. Based on our review of four after-action reports issued by  HSC between April 2007 and August 2008, three did not meet program  requirements.  In one case, the after-action report was drafted but not  finalized.  In two other cases, the report did not identify officials who were  responsible for ensuring corrective actions were implemented.  For  example, the after-action report for the February 2008 Principal Level  Exercise on pandemic influenza stated that the integration of strategic  communications and policy remains difficult and should be addressed by  an interagency group.  However, the after-action report did not identify a  department or an agency official who was to be responsible for ensuring  that this corrective action was implemented.  FEMA officials said they  relied on the National Exercise Program Implementation Plan as the  guiding policy for HSC’s responsibilities for documenting after-action  reports and tracking and resolving corrective actions.  However, the plan  does not describe a procedure for ensuring that these requirements are  met by HSC.  Based in part on our review of Principal Level Exercises,  FEMA has subsequently taken action to identity officials who are  responsible for nearly all of the corrective actions outlined in these after- action reports.
The implementation plan for the National Exercise Program also requires  entities responsible for Tier I level exercises, including Principal Level  Exercises, to ensure that corrective actions are resolved.  However, HSC  did not follow the required corrective action process.  According to HSC  staff, the council does not use the CAP system or another tracking  procedure for determining whether corrective actions were implemented.   Rather, it delegates the responsibility of taking corrective actions to the  appropriate agencies or departments.
FEMA officials agreed that tracking corrective actions for Principal Level  Exercises is problematic and impairs their ability to fulfill FEMA’s  statutory obligation to track corrective actions from exercises.  However,  they stated that they do not have the authority to direct HSC or other  federal entities to track corrective actions and report this information to  FEMA.  FEMA officials said that they rely on the National Exercise  Program Implementation Plan to set forth HSC’s responsibilities for  documenting after-action reports and tracking and resolving corrective  actions.
Although the Post-Katrina Act does not give FEMA the authority to compel  other federal entities to comply with the objectives of the National  Exercise Program, the act places responsibility for implementing the  National Exercise Program on FEMA, in coordination with other  appropriate federal entities.  Therefore, it is incumbent on FEMA to  coordinate with HSC and other federal entities to better ensure that FEMA  obtains the information it needs to meet its statutory responsibility to  track corrective actions.  In this regard, the implementation plan for the  National Exercise Program does not set forth FEMA’s statutory  responsibility to track corrective actions, nor does it require federal  departments and agencies to report corrective action information to  FEMA.  On the contrary, the implementation plan provides that  departments and agencies “may submit issues to the DHS Corrective  Action Program (DHS CAP) through the web-based DHS CAP system,” but  does not instruct them to do so or to otherwise provide FEMA with  corrective action tracking information that would enable FEMA to fulfill  its statutory responsibilities under the Post-Katrina Act.  Therefore, the  implementation plan lacks procedures that call on HSC and other federal  entities to report corrective action information to FEMA.  FEMA’s inability  to fully track and analyze areas that need improvement is also due, in part,  to its lack of an effective internal controls environment.  GAO’s Standards  for Internal Controls in the Federal Government state that an effective  internal control environment is a key method to help agency managers  achieve program objectives and enhance their ability to address  weaknesses.  The standards state, among other things, that agencies  should have policies and procedures for ensuring that the findings of  audits and reviews are promptly resolved.  The standards also state that  internal controls should generally be designed to assure that ongoing  monitoring occurs.   Developing procedures for working with federal  entities, such as HSC, to help ensure that corrective actions are tracked,  implemented, and reported to FEMA would strengthen FEMA’s ability to  determine emergency management areas that need improvement.
Lessons learned from Hurricane Katrina identified similar concerns with  tracking corrective actions from exercises.  For example, in February 2006  the White House report on Hurricane Katrina stated that “too often, after- action reports for exercises and real-world incidents highlight the same  problems that do not get fixed”.  According to the report, DHS should  ensure that all federal and state entities are accountable for the timely  implementation of remedial actions in response to lessons learned.   According to the report, the success of the preparedness system depends,  in part, on feedback mechanisms for tracking corrective actions.  When  federal entities carry out processes that are incompatible with FEMA’s  responsibilities for tracking corrective actions, FEMA managers do not  have the necessary data to measure progress, identify gaps in  preparedness, and track corrective actions—key objectives of the National  Exercise Program.
section_title: FEMA Does Not Have Procedures to Help Ensure That After-Action Reports from State-Level Exercises Are Properly Prepared and Submitted and That Corrective Actions Are Tracked and Implemented
paragraphs: Similar to the problems we found with Principal Level Exercises, we  identified weaknesses in the way in which selected states prepared and  submitted after-action reports to FEMA.  Among other things, HSEEP  requires that exercise program managers prepare after action-reports that  include improvement plans to identify corrective actions, track whether  the actions were implemented, and continually monitor and review  corrective actions as part of an organizational corrective action program.   Exercise program managers are to submit these after-action reports to  FEMA through its Secure Portal—a FEMA database containing after- action reports.  Of the six states we visited, (1) none systematically  recorded or submitted after-action reports to FEMA, (2) only one had  improvement plans in all of its after-action reports and only one said it had  a corrective action tracking program, and (3) none used a capabilities- based approach in all of their exercises.  The following are additional  details on each of these issues.
HSGP guidance requires exercise program managers to submit after- action reports within 60 days following the completion of an exercise  to FEMA through its Secure Portal.  Although the portal has been  operational for about 5 years, FEMA did not have procedures in place  to fully monitor state actions and to ensure that this occurred.  While  3 of 44 after-action reports provided by officials from the six states we  visited were submitted to FEMA in the requisite area of the portal,  these reports all came from one state.  The remaining five states did not  submit any after-action reports through the portal.  Officials from these  five states cited technical difficulties, lack of staff resources, or unclear  guidance from FEMA as reasons why after-action reports were not  submitted to the portal.   FEMA is aware that the portal contains  incomplete information, noting in its quarterly newsletter that “The  Secure Portal serves as the repository for after action reports and  improvement plans; however, postings have been inconsistent.  At  times, it has been difficult to locate After Action Reports as many are  posted in draft form and never finalized and posted outside the  respective State folder.”   In February 2009 FEMA announced, a  National Exercise Division Exercise Support System that is an online  tool for facilitating exercise planning to replace the Secure Portal as a  repository for after-action reports.
HSEEP requires that entities include an improvement plan as part of  their after-action reports and that they have a corrective action  program.  While each of the six states we visited had produced at least  one draft after-action report that included an improvement plan, only  one state included an improvement plan in all of its reports. Fifteen of  the 44 after-action reports we reviewed had an improvement plan.  In  addition, only one state had a corrective action program that tracked  whether corrective actions were implemented.  Officials from one state  attributed the lack of a corrective action program to competing  priorities.  Specifically, states are involved in many exercises and  officials are more likely to place the priority on designing and  conducting of the next exercise than on tracking corrective actions  from prior exercises.
Another HSEEP requirement calls for exercises to be designed and  conducted using a capability-based approach.  Doing so would help  FEMA analyze whether gaps in capability have narrowed and  improvements in capabilities have occurred from the use of grant funds  by states.   However, 20 of the 44 after-action reports provided by  officials in the six states we visited used target capabilities, while the  remaining 24 did not.   According to officials from three of the six  states we visited, not all exercise participants have a good  understanding of target capabilities and how they should be used in the  design, conduct, and evaluation of exercises.  For example, exercise  officials from one state said their state does not use target capabilities  because it has its own set of assessment standards.
FEMA’s lack of procedures for monitoring states to ensure compliance  with HSEEP requirements contributed to limited adherence to such  requirements in the states we visited.  As discussed earlier in this report,  internal control standards call for (1) an effective internal control  environment to help agency managers achieve program objectives and  enhance their ability to address weaknesses, (2) agencies to have policies  and procedures for ensuring that the findings of audits and reviews are  promptly resolved, and (3) internal controls to generally be designed to  assure that ongoing monitoring occurs.  FEMA officials in the National  Preparedness Directorate told us they have a process for monitoring  HSEEP compliance by, among other things, having FEMA regional  exercise support program managers discuss HSEEP compliance with state  exercise program officials at planning conferences or during grant  monitoring discussions.  While discussing HSEEP requirements at annual  conferences may enhance the awareness of state officials about  requirements for HSEEP compliance, this does not track compliance.  In  addition, officials from FEMA’s Grant Programs Directorate said they do  not monitor states’ compliance with HSEEP requirements.  For example,  the grant monitoring reports for the six states we visited did not address  whether the states were in compliance with HSEEP requirements.  Such  reports are based, in part, on a checklist of items that officials use to  monitor compliance with grant requirements.  However, FEMA’s checklist  does not include specific items, such as compliance with HSEEP  requirements, as called for by HSGP guidance.  States’ noncompliance  with HSEEP hinders their ability to systematically track corrective actions  and assess capabilities.  This in turn impacts FEMA’s ability to measure  progress of the National Exercise Program.  Having procedures in place to  monitor actions by states to ensure compliance with HSEEP requirements  would assist FEMA in obtaining more complete data about the results of  exercises and corrective actions taken to systematically evaluate readiness  through the National Exercise Program, as required by the Post-Katrina  Act.
section_title: FEMA’s Ability to Design Exercises That Stress the Preparedness System Is Limited
paragraphs: The Post-Katrina Act requires FEMA to stress the preparedness system  through the National Exercise Program to evaluate preparedness for a  catastrophic event.  The National Exercise Program Implementation Plan  identifies domestic incident management for catastrophic events as the  principal focus of the National Exercise Program.  According to the Tier I  exercise cycle established in the implementation plan, FEMA plans to test  for a catastrophic domestic nonterrorism event in fiscal year 2010.   However, FEMA’s ability to meet this testing requirement is limited by  three factors:  (1) the lack of key planning documents, (2) exercise  artificiality, and (3) limited coordination with groups that have expertise in  populations with special needs.
First, the effectiveness of exercises is based, in part, on the degree to  which plans that define roles and responsibilities have been developed.
The fact that key planning documents for response to a catastrophic  incident, such as the supplement to the catastrophic incident annex and  regional response plans, have not yet been completed means that the  National Exercise Program will have difficulty in designing exercises that  test whether the plans are understood and executed effectively by  stakeholders.88, 89  As we described earlier in this report, while DHS and  FEMA are working on these plans, it is unclear when they will complete  the plans.  According to the former director of FEMA’s National  Preparedness Directorate, FEMA’s ability to design and conduct exercises  that evaluate a response to a catastrophic incident is limited by the fact  that plans such as those described above have yet to be developed.   However, the official indicated that FEMA was taking preliminary actions  to build its capacity to conduct such exercises through regional  catastrophic planning initiatives.
The supplement to the catastrophic incident annex is required by the Post-Katrina Act.  6  U.S.C. § 319(b)(2)(C). While a version of the supplement was written before the act, an  updated supplement has yet to be published.
Regional plans have been drafted; however, according to FEMA officials, the plans are not  consistent from one region to another because regions developed plans without any  guidance from FEMA headquarters.
Exercise Program’s Implementation Plan, such participation from other  federal stakeholders is not required for Tier II exercises such as the one  held in May 2008.
Third, another challenge in creating exercises that stress the preparedness  system and simulate real-world conditions is finding ways to test response  capabilities for populations with special needs.  To address some of the  problems experienced in Hurricane Katrina in dealing with populations  with special needs, such as residents in nursing homes, the Post-Katrina  Act, as amended by the 9/11 Act, called on FEMA to design exercises to  address the unique requirements of populations with special needs,  including the elderly, and to coordinate the National Exercise Program  with the National Council on Disability, among other entities.  In  TOPOFF 4, FEMA integrated specific objectives for special needs  populations in the Oregon venue, according to FEMA officials.  For  example, according to FEMA officials, FEMA utilized special needs actors  to enhance realism.  However, HSEEP guidance does not address special  needs populations. Further, while FEMA has corresponded with the  Council on Disability, council officials believe that FEMA could do more  to ensure that exercises are designed to address the unique requirements  of populations with special needs.  For example, council officials stated  that the council was not involved in the design and planning for TOPOFF 4  or the May 2008 Tier II exercise.  According to officials from the National  Preparedness Directorate, it coordinated with FEMA's Special Needs  Office to integrate special needs population objectives into exercises.   FEMA officials agree that special needs populations should be included in  exercises and they said that they will redouble their efforts to do so.   However, FEMA officials also noted that some exercises, for example, the  National Level Exercise planned in July 2009, may not involve special  needs populations because the point of such exercises is to prevent a  terrorist attack, rather than to test response and recovery efforts.  Enhancing coordination with the National Council on Disability could  improve FEMA’s ability to ensure that key issues concerning populations  with special needs are addressed in the design and conduct of exercises.
The limitations the National Exercise Program faces in designing  approaches that stress the preparedness system highlight the difficulty in  validating whether roles and responsibilities are well understood and  major gaps in capabilities remain for responding to and recovering from  catastrophic events.  The 2006 White House report on the federal response  to Katrina concluded that the “national preparedness system must be  oriented toward greater challenges.  We must not shy away from creating  scenarios that stress the current system of response to the breaking  point……Until we meet the standards set by the most demanding  scenarios, we should not consider ourselves adequately prepared.”   In  2006, we reported that effective exercises should involve scenarios that  stress responders with the highest degree of realism possible, even to the  breaking point if possible.  Exercises that stress the preparedness system  in a realistic way are key to testing the prospective reliability of a response  and determining whether plans have accounted for potential breakdowns  with relatively greater consequences.
In February 2009, we met with FEMA officials to discuss this issue and  they agreed that developing exercises to the point of system failure is a  valid objective; however, they described several factors that may limit  their ability to do so.  For example, FEMA officials told us that exercising  to the “breaking point” requires significant resources that under current,  and likely future funding streams, are unlikely to be available.  We agree  with FEMA that these are important considerations; however, these  considerations, in part, are addressed through the implementation plan for  the National Exercise Program, which describes a 5-year schedule of  exercises to give federal departments and agencies lead time to budget for  participation in such events.
section_title: FEMA’s Data for Measuring the Effectiveness of Its National Exercise Program Is Incomplete
paragraphs: The three databases that FEMA uses to measure the effectiveness and  progress of the National Exercise Program have incomplete data.  FEMA  uses (1) the NEXS system to identify DHS-funded exercises, (2) the FEMA  Secure Portal as a repository for DHS funded exercise after-action reports,  and (3) the CAP system as a tool for tracking corrective actions.  The  following provides details on problems with the reliability of each of these  databases.
FEMA calls on state entities to use the NEXS system to schedule all  exercises, and one of the performance measures that FEMA uses to  assess and report on the performance of the National Exercise  Program is the number of DHS-funded state exercises that occur per  year.  However, 26 of the 44 after-action reports we reviewed did not  have the exercise entered into the NEXS system.  Incomplete NEXS  system data limit FEMA’s ability to accurately report on the number of  DHS-funded exercises. Furthermore, while FEMA created the NEXS  system to schedule, synchronize, and avoid conflicts in all national,  federal, state, and local exercises, it cannot do so with incomplete data.   When we discussed this problem with state and FEMA officials, they  agreed that the NEXS system did not contain a comprehensive list of all  state and local exercises supported by HSGP funds.  For example,  when we asked FEMA for a complete list of all exercises to be  conducted under the National Exercise Program, the agency could not  produce such a list.  Standards for Internal Control in the Federal  Government state that program managers need data to determine  whether they are meeting their performance targets and that controls  should be designed to validate the integrity of organizational  performance measures and indicators.  FEMA has initiated actions to  validate the accuracy of data used in the NEXS system by using training  and exercise plan workshops with states to determine what exercises  states have scheduled.  We agree that the training and exercise plan  workshops are a good starting point for verifying the completeness of  NEXS; however, when we attended a training and exercise plan  workshop, we were told by FEMA officials that not all federal agencies  or local entities participated, thus, all exercises were not discussed at  the workshop.  In addition, FEMA officials recognize that the  workshops alone do not ensure that the NEXS database is complete  and accurate.  In the absence of systematic and comprehensive  information on the number of federally funded exercises, FEMA cannot  measure its progress in implementing the National Exercise Program.
A second database used by FEMA is the Secure Portal—the primary  database that FEMA uses to measure the degree to which states  comply with HSEEP.  Even though FEMA requires state exercise  program managers to place their after-action reports in the Secure  Portal when federal grant funds are used to support the exercise, this  requirement was not completely met by any of the six states we visited.   In addition, although state and FEMA officials agreed that the Secure  Portal does not contain all state exercise after-action reports, FEMA  uses information from the portal to assess and report on the  performance of the National Exercise Program, including the measure  that FEMA uses to assess the percent of DHS-funded exercises  demonstrating the use of HSEEP guidance. Since the Secure Portal  contains incomplete information and neither FEMA nor the six states  we visited have controls to ensure that all state-level exercise after- action reports are uploaded to the Secure Portal, this measure may not  accurately reflect the percent of DHS-funded exercises that  demonstrate the use of HSEEP guidance.
Third, although federal entities involved in Tier 1 exercises are  encouraged to use FEMA’s CAP system, it does not contain all  corrective actions from such entities.  According to FEMA officials, the  CAP system was designed to capture all relevant and necessary  information related to the implementation of corrective actions.   However, the HSC did not use the CAP system to track corrective  actions.  The problem with having incomplete data in the CAP system  is that FEMA uses information from the system to measure the  percentage of corrective actions that have been implemented as one of  its performance measures.  A key reason for this problem is that  FEMA has not established procedures to ensure that the information in  the CAP system is complete.  For example, the implementation plan  does not require the nine federal departments and agencies that are  signatories to the implementation plan to use the CAP system.  Instead,  FEMA strongly urges stakeholders to use the system, but the decision  to do so is discretionary.  FEMA officials cited the tension between  requiring entities to use the CAP system and providing enough  flexibility to those entities to carry out their programs as a reason for  not making the use of the CAP system a requirement.  Nonetheless,  entities could submit a report to FEMA on the status of their corrective  actions resulting from such exercises.
Finally, the CAP system does not include corrective actions from real- world incidents and FEMA has not established requirements or guidelines  for agencies to do so.  As a result, FEMA is unable to meet Post-Katrina  Act requirements for conducting long-term trend analyses of corrective  actions that include real-world events.  The Presidential Inaugural  Ceremony held in Washington D.C. on January 20, 2009, provides an  example of the importance of tracking corrective actions for real-world  events.  During the event, problems with managing crowds prevented a  large number of ticket holders from reaching their designated area to  observe the inauguration ceremony. According to the Joint Congressional  Committee on Inaugural Ceremonies, a complete examination will take  place to provide a foundation of lessons learned for future inaugural  planners, so that they can avoid similar problems in the future.  FEMA  officials agreed that the CAP system could be used to track corrective  actions from real-world events, and in February 2009 they indicated that  developing procedures to do so would aid their ability to conduct long- term trend analyses of real-world events as required by the Post-Katrina  Act.
section_title: FEMA Has Taken Actions to Develop and Implement a Comprehensive System for Assessing National Preparedness Capabilities, but Faces Challenges in Completing and Implementing the System
paragraphs: FEMA has taken initial actions to collect information on state  preparedness capabilities and develop a comprehensive assessment  system for assessing capabilities at all levels of government, but faces  methodological and coordination challenges in completing the system.   Assessing and reporting on national preparedness is a long-standing and  complex effort that presents methodological, integration, and coordination  challenges.  Effectively addressing these challenges requires that FEMA  take a measured and planned approach; however, FEMA’s project  management plan does not fully identify the numerous program elements  and how and when they will be developed and integrated.
subsections: section_title: FEMA Has Taken Initial Actions to Develop a Comprehensive Assessment System and Collect Information for State and Federal Capability Reporting
paragraphs: The Post-Katrina Act requires that FEMA establish a comprehensive  assessment system to assess the nation’s capabilities and overall  preparedness for preventing, responding to, and recovering from natural  and man-made disasters.  The act also requires that FEMA collect  information on state capability levels and report on federal preparedness  to Congress, including, among other things, the results of the  comprehensive assessment system.  In response to these requirements,  FEMA established guidance for reporting on state preparedness and  created the Office of Preparedness Policy, Planning and Analysis (PPPA)  to develop and implement a new assessment approach that considers past  efforts and integrates ongoing assessment efforts.  FEMA plans to  integrate the state preparedness reports—along with a variety of existing  assessment efforts and data sources—into the new comprehensive system  it is establishing.  In addition, it is considering the historical experiences  and lessons learned from prior assessment efforts in developing the new  system.
FEMA has also made progress in collecting information for federal and  state reporting.  In January 2009, FEMA issued its first federal  preparedness report.  The Post-Katrina Act requires that FEMA, in  coordination with the heads of appropriate federal agencies, submit a  federal preparedness report to Congress beginning in October 2007 and  annually thereafter, which is to include, among other things, the results of  the comprehensive assessment.  In addition, states, territories, and the  District of Columbia completed and submitted their first state  preparedness reports to FEMA in the spring of 2008—a total of 56 reports  from the 56 jurisdictions receiving homeland security grant funding.   FEMA officials said they prepared summaries of the 56 reports and  provided the summaries to FEMA’s regional offices.  In addition, in  November 2008, FEMA issued guidance for the 2008 state preparedness  reports, which grantees are to submit to FEMA by March 2009.
section_title: FEMA Faces Methodological and Coordination Challenges in Completing the Comprehensive Assessment System and Reporting on Its Results
paragraphs: FEMA faces methodological challenges with the four assessment systems  it plans to use as the basis for the new system and has not determined how  to overcome problems faced in historical assessments.  The challenges  FEMA faces reflect the lack of guidance from PPPA in how the assessment  system will comprehensively inform and incorporate feedback from other  elements of the National Preparedness System and information from a  variety of other data sources.  Finally, FEMA faces challenges in  coordinating with state, local, and federal stakeholders in developing and  implementing the system and reporting on its results.
subsections: section_title: Methodological Challenges with Using Four Proposed Assessment Approaches
paragraphs: In December 2008, FEMA provided us with a project management plan  outlining efforts to establish the comprehensive assessment system by  May 2010 to “function as a central repository for national preparedness  data.”  The system “will integrate data from prior reports and legacy  assessment systems.” To establish the system, FEMA plans to administer a  Web-based survey to all states and territories in the summer of 2009 to  assess capabilities using the 37 target capabilities.  FEMA plans to use a  Web-based system known as the National Incident Management System  Compliance Assessment Support Tool (NIMSCAST) to administer the  capability assessment survey.  In addition, FEMA noted that NIMSCAST  will serve as the technical foundation for the comprehensive assessment  system, and that the system is used by all states and territories as well as  by 18,000 local and tribal entities, which helps to mitigate challenges  FEMA faces in coordinating with stakeholders in developing and  implementing the comprehensive assessment system.  However, FEMA  faces methodological challenges with regard to (1) differences in data  available, (2) variations in reporting structures across states, and (3)  variations in the level of detail within data sources requiring subjective  interpretation, as summarized in table 2 below.  Additional information  regarding these assessments is outlined in appendix III.
In addition to the methodological challenges in its current approach to  assessing capabilities, previous efforts at assessing capabilities  experienced challenges and have been discontinued.
The National Preparedness System was discontinued by DHS officials  because it was time-consuming and did not produce meaningful data,  according to FEMA officials.  The system was pilot tested in 10 states  and according to budget documentation, FEMA spent nearly $15  million in total for 2006, 2007, and 2008 on the system before it was  discontinued.
The Pilot Capability Assessment was labor-intensive and did not  generate meaningful data, according to FEMA officials.  This  assessment, piloted in six states, was intended to measure  jurisdictions’ progress in achieving needed target capabilities.   Because it was only piloted, FEMA did not generate meaningful  preparedness information from the data collected, according to  officials.
The Capability Assessment for Readiness, which was proposed as a  one-time nationwide assessment of capabilities, lacked controls for  validating the accuracy of self-reported assessment data.  The  assessment was conducted in 1997 but concerns regarding self- reporting and the lack of controls for validating information reported  by states limited the reliability and, therefore, the value of the data,  according to the DHS Inspector General.
Additional information regarding these efforts is outlined in appendix III.
section_title: FEMA Faces Integration Challenges in Completing the Comprehensive Assessment System
paragraphs: FEMA has not established an approach for how information and data from  different sources will be integrated into the comprehensive assessment  system.  FEMA officials have established a charter between the National  Preparedness Directorate and Grant Programs Directorate to coordinate  preparedness efforts related to the Cost to Capability Initiative and  refinement of the target capabilities.  FEMA has also begun sharing  information between staff involved in developing the assessment system  and staff involved in other elements of the National Preparedness System.   For example, staff from the National Exercise Program said they shared  information on their exercise efforts with staff developing the  comprehensive assessment system and FEMA officials said they  established a working group of officials from other federal agencies to  communicate efforts to develop the comprehensive assessment system.  In  March 2009, FEMA officials acknowledged that they had not finalized a  charter for this working group to outline the specific actions that the  working group will undertake to develop the comprehensive assessment  system.  FEMA explained that this working group will (1) identify existing  sources of data related to preparedness plans, organization, equipment,  training, and exercises; (2) vet the relevancy of each data source for  assessments; (3) identify data gaps and redundancies; and, (4) develop  recommendations for streamlining data collection and reporting.   However, FEMA has not established an approach for integrating  information and data from other stakeholders, including grantees and  other FEMA divisions such as the Disaster Operations Directorate.
In October 2008, FEMA officials said they also plan to consider or  incorporate into the new system a multitude of other data and analysis  sources within and outside of DHS, such as FEMA’s Biannual Strategy  Implementation Reports; Homeland Security Grant Program Investment  Justification; and Tactical Interoperable Communications Plan  Scorecards.  In addition, FEMA plans to use the CAP system and LLIS to  inform the comprehensive assessment system.  In February 2009, FEMA  officials further explained that they plan to rely on three indicators of  preparedness to develop the comprehensive assessment system: (1) state  and federal preparedness reports, which are required to use target  capabilities; (2) the results of exercise corrective action findings; and (3)  operational plans outlining specific operational requirements for all levels  of government.  However, FEMA has not established an approach for how  the three indicators of preparedness will be collected and developed into  reporting mechanisms that meet Post-Katrina Act requirements for the  comprehensive assessment system.  In its first federal preparedness  report, FEMA acknowledged that its efforts to evaluate and improve  preparedness are the least mature elements of the national preparedness  system because these efforts are composed of a wide range of systems and  approaches with varying levels of integration.  Given the relative  immaturity of FEMA’s evaluation and improvement efforts, without an  approach for integrating its comprehensive assessment system efforts,  FEMA faces increased risks that inconsistencies may occur or that data  and information are not shared, limiting FEMA’s ability to fulfill the  requirements of the Post-Katrina Act for developing the system.
section_title: FEMA Faces Coordination Challenges in Establishing Quantifiable Metrics and Collecting Information for Implementing the Comprehensive Assessment System
paragraphs: In addition to methodological and coordination challenges in developing  and completing the comprehensive assessment system, FEMA faces  coordination challenges in establishing quantifiable metrics for target  capabilities outlined in the Target Capabilities List.  Establishing  quantifiable metrics for target capabilities is a prerequisite to developing  assessment data that can be compared across all levels of government.    At the time of our review, FEMA was in the process of refining the target  capabilities to make them more measurable and to provide local and state  jurisdictions with additional guidance on the levels of capability they need.   FEMA plans to develop quantifiable metrics—or performance objectives— for each of the 37 target capabilities that are to outline specific capability  “targets” that jurisdictions, including but not limited to cities, of varying  size should strive to meet.  FEMA plans to complete quantifiable metrics  for all 37 target capabilities by the end of 2010.  As of February 2009,  FEMA noted that 6 of the 37 capabilities were undergoing stakeholder  review and that FEMA planned to develop quantifiable metrics for a total  of 12 capabilities during the 2009 calendar year.  However, as of March  2008, FEMA had not developed milestones for completing quantifiable  metrics for the remaining 25 target capabilities.  Cognizant of the fact that  there is not a “one size fits all” approach to preparedness, FEMA also plans  to develop performance classes for each target capability in order to  account for differences in levels of preparedness across jurisdictions of  varying size and risk.  FEMA plans to incorporate the new performance  objectives and performance classes into the comprehensive assessment  system, the federal preparedness report, and the guidance for the state  preparedness reports, but has not established a time frame for doing this.
FEMA recognizes the need to coordinate with federal, state, and local  stakeholders and ensure that their views are effectively integrated in the  development of the metrics, but FEMA historically has faced challenges in  coordinating with stakeholders.  For example, as we reported in June  2008, FEMA’s efforts to coordinate with stakeholders in developing the  NRF were inconsistent and needed to be improved.  In addition, such  coordination can be time consuming.  For example, in July 2005 we  reported on DHS’s prior effort to develop a tiered system of metrics based  on population density and critical infrastructure in order to (1) assign  jurisdictions responsibility for developing and maintaining target  capability levels and (2) use the metrics to implement a national “balanced  investment program” (with the purpose of directing federal preparedness  assistance to the highest priority capability gaps) for national  preparedness capabilities.  DHS scheduled this system to be developed  and completed by October 2008.  At the time of our review, these efforts  to develop quantifiable metrics for target capabilities were not complete,  illustrating the fact that developing metrics in coordination with a variety  of stakeholders can take longer than anticipated.  In developing the  quantifiable capability metrics, FEMA officials told us that they plan to  conduct extensive coordination with stakeholders that will entail  conducting stakeholder workshops in all 10 FEMA regions and  coordinating with all federal agencies with lead and supporting  responsibility for Emergency Support Function (ESF) activities associated  with each of the 37 target capabilities.  Officials said they also plan on  briefing the National Advisory Council and the National Council on  Disability, and soliciting public comment on the draft quantifiable metrics  for each target capability.
One of FEMA’s coordination efforts—working with nonfederal  stakeholders and federal agencies responsible for ESF activities— illustrates the large number of stakeholders with whom FEMA plans to  coordinate in developing quantifiable metrics for the target capabilities.   FEMA also plans to post each revised capability to the Federal Register  for comment.  With respect to federal agency coordination, FEMA plans  to coordinate with each federal agency that has lead and supporting  responsibility for ESF activities associated with each of the 37 target  capabilities in developing quantifiable capability metrics.  For example,  for the medical surge capability, the Department of Health and Human  Services (HHS) is the primary federal agency responsible for coordinating  necessary medical surge capabilities needed during a disaster to provide  triage and medical care services.  In addition to HHS, 15 other federal  agencies and the American Red Cross are designated as supporting  agencies and organizations for medical surge capabilities.  Coordinating  with other federal agencies responsible for ESF activities in developing  quantifiable capability metrics would likely entail time, effort, and  unforeseen risks.  For example, in September 2008 we reported on the  risks that FEMA faces in coordinating with external stakeholders, namely  the American Red Cross, in collecting and integrating preparedness  information necessary to develop the comprehensive assessment  system. In that report, we recommended that FEMA take steps to better  incorporate information from voluntary organizations related to sheltering  and feeding capabilities, which are elements of the mass care target  capability, and noted that a comprehensive assessment of the nation’s  capabilities should account as fully as possible for the voluntary  organizations’ capabilities in mass care.  FEMA disagreed with the  recommendation, noting that it cannot control the resources of nonprofit  and private organizations.  In response, we (1) noted that taking steps to  assess capabilities more fully does not require controlling these resources,  but rather cooperatively obtaining and sharing information and (2)  reiterated that such efforts are important for assessing the nation’s  prevention capabilities and overall preparedness.
FEMA’s efforts to collect information needed to draft and issue the first  federal preparedness report, required by the Post-Katrina Act, also reflect  the coordination challenges FEMA faces in implementing the  comprehensive assessment system.  The Post-Katrina Act requires that  FEMA, in coordination with the heads of appropriate federal agencies,  submit a federal preparedness report to Congress beginning in October  2007 and annually thereafter, which is to include, among other things, the  results of the comprehensive assessment.  FEMA issued the first federal  preparedness report in January 2009.  In response to our comment that the  draft report had been under review for 8 months between March 2008 and  November 2008, FEMA noted that after completing a review of the report  by FEMA and DHS, the report was submitted to the Office of Management  and Budget to disseminate to all federal departments and agencies for  review and comment. Officials explained that in developing the report,  they faced challenges in obtaining information and data from federal  agencies because of bureaucratic obstacles for collecting information and  also faced challenges in analyzing information from multiple sources.   FEMA officials said they may develop a National Preparedness Report to  combine two Post-Katrina Act reporting requirements—the requirement  for an annual federal preparedness report and an annual catastrophic  resources report—and to include information from state preparedness  reports as part of this consolidated report, which they tentatively plan to  issue in the Spring of 2009.
section_title: FEMA Has Not Developed an Approach and Assessed Risks for Completing and Implementing the Comprehensive Assessment System as Part of Its Project Management Plan
paragraphs: Despite the methodological and coordination challenges associated with  developing a new comprehensive assessment system and establishing  related quantifiable metrics for target capabilities, FEMA has not  developed an approach that addresses program risks as part of its project  management plan for how it will develop the comprehensive assessment  system.  While FEMA has developed a project management plan for  completing the comprehensive assessment system by 2010, the lack of (1)  milestones for establishing quantifiable metrics for all 37 target  capabilities and (2) specific actions for how FEMA will integrate  preparedness information to develop the system, coupled with the (3) the  lack of risk assessment information for the system raises questions about  FEMA’s ability to establish the system in accordance with its anticipated  2010 completion date.  FEMA has described several steps for completing  an effective comprehensive assessment system that include developing  methodologies to translate the information from the assessments FEMA  has identified into a target capabilities-based framework and integrating  necessary preparedness information from disparate and not necessarily  comparable sources such as state preparedness reports.  FEMA also plans  to coordinate with relevant stakeholders to refine the target capabilities.
However, FEMA’s plan does not outline specific actions it plans to take to  do so.
Certain factors, such as challenges in coordinating with stakeholders or  difficulties in obtaining necessary data, could affect FEMA’s ability to  implement the comprehensive assessment system.  Best practices for  project management established by the Project Management Institute state  that managing a project involves project risk management, which serves to  increase the probability and impact of positive events, and decrease the  probability and impact of events adverse to the project.  Project risk  management entails determining which risks might affect a project,  prioritizing risks for further analysis by assessing their probability of  occurrence, and developing actions to reduce threats to the project.  Other  practices include (1) establishing clear and achievable objectives; (2)  balancing the competing demands for quality, scope, time, and cost; (3)  adapting the specifications, plans, and approach to the different concerns  and expectations of the various stakeholders involved in the project; and  (4) developing milestone dates to identify points throughout the project to  reassess efforts underway to determine whether project changes are  necessary.  FEMA has demonstrated its awareness of the value of these  practices.  For example, in planning another project—an effort to  transition the FEMA Secure Portal to an alternate information technology  platform known as the National Exercise Division Exercise Support  System —FEMA identified key elements such as phases, milestones, and  risks that could affect the project goals.
Furthermore, a risk assessment could help FEMA define the specific  actions to take to complete the comprehensive assessment system,  anticipate potential delays in completing its efforts to refine the target  capabilities by 2010, and deal with the associated risks in its efforts to do  so, such as the time it takes to coordinate with stakeholders.  Information  from a risk assessment could also enhance FEMA’s ability to coordinate  with federal agencies to obtain preparedness information needed to  produce a timely annual federal preparedness report and catastrophic  resources report.  Until FEMA assesses ways to mitigate the risks  associated with its capability assessment efforts, it will be difficult for  FEMA to provide reasonable assurance that it can produce a  comprehensive assessment system that (1) fulfills the requirements of the  Post-Katrina Act and in the long term, (2) informs decisions related to  improving national preparedness.
section_title: FEMA’s Implementation of the National Preparedness System Would Be Enhanced by a Strategic Plan That Includes Desirable Characteristics
paragraphs: FEMA’s National Preparedness Directorate does not have a strategic plan  for implementing the national preparedness system.  The complexity and  difficulty of implementing the national preparedness system, which we  describe earlier in this report, underscore the importance of strategic  planning.  The six desirable characteristics of a national strategy can help  the National Preparedness Directorate in developing a strategic plan.
subsections: section_title: FEMA’s National Preparedness Directorate Does Not Have a Strategic Plan for Implementing the National Preparedness System
paragraphs: While FEMA has recognized that its components need to develop strategic  plans that detail program goals, objectives, and strategies, FEMA’s  National Preparedness Directorate (Preparedness Directorate) has not yet  developed such a plan for the national preparedness system.  In January  2008, FEMA issued its agencywide strategic plan, which set a common  direction for its components in carrying out their responsibilities in  preparedness, response, and recovery programs.   While a  Preparedness Directorate official acknowledged that the Preparedness  Directorate does not have a strategic plan (this is the responsibility of  FEMA’s Office of Preparedness Policy, Planning and Analysis), the official  said the Post-Katrina Act provides a roadmap that contains the  preparedness strategy and FEMA uses an annual operating plan (in draft  form at the time of our review) to guide the directorate’s approach for  implementing the national preparedness system.
Although the Post-Katrina Act and the directorate’s draft annual operating  plan outline certain elements of a strategy, such as the directorate’s vision,  mission, and goals, they do not include several other desirable  characteristics of a strategic plan, such as a discussion of how the  directorate will (1) measure its progress in developing the national  preparedness system, (2) address risk as it relates to preparedness  activities, (3) coordinate with its preparedness stakeholders in developing  and carrying out the various elements of the national preparedness  system, and (4) integrate the elements of the national preparedness  system.  For example, FEMA has not included information on  performance measures for meeting one of the objectives outlined in the  operating plan—to support an integrated planning system for the federal  preparedness-related agencies that links to regional, state, and local  planning activities.  The operating plan also does not define the problem or  assess the risks that FEMA’s national preparedness program faces.   Specifically, it does not describe the threats, vulnerabilities, and  consequences of a major homeland security incident or what FEMA’s  approach will be for addressing risk through its national preparedness  system activities.  While the draft operating plan identifies subcomponents  in the Preparedness Directorate that will be responsible for carrying out  segments of the national preparedness system, it does not discuss the  roles and responsibilities of preparedness stakeholders, the coordination  that will occur between them, or how the four elements of the national  preparedness system will be integrated.
The Post-Katrina Act calls on FEMA to develop and coordinate the  implementation of a strategy for preparedness, and the complexity and  difficulty of developing a national preparedness system underscore the  importance of strategic planning.   An important element of strategic  planning is that it presents an integrated system of high-level decisions  that are reached through a formal, visible process.  The resulting strategy  is thus an effective tool with which to communicate the mission and  direction to preparedness stakeholders.  The conditions we describe  earlier in this report—such as incomplete plans on roles and  responsibilities, unresolved corrective actions from exercises, and  potential difficulties and historical delays in capability assessments—show  that FEMA faces significant challenges in developing the key elements of  the national preparedness system.
section_title: The Desirable Characteristics of a National Strategy Can Help FEMA’s National Preparedness Directorate in Developing a Strategic Plan
paragraphs: In 2004, we identified six desirable characteristics of an effective national  strategy that help achieve strategy success.  These characteristics are  summarized in table 3.
We believe these characteristics can assist responsible parties, such as  FEMA, in further developing and implementing national strategies as well  as enhancing these strategies’ usefulness for policy decisions to help  achieve program results and accountability.  The characteristics are a  starting point for developing a strategic plan.  However, we believe that an  approach incorporating the substance of these characteristics is likely to  increase success in strategy implementation.  The following describes how  each of these characteristics applies to the work of FEMA’s National  Preparedness Directorate.
Purpose, scope, and methodology: National preparedness is an  important part of homeland security efforts outlined in legislation,  presidential directives, and policy documents.  The National Strategy for  Homeland Security recognized the importance of fostering a Culture of  Preparedness that permeates all levels of our society, including all  preparedness stakeholders.   In summarizing lessons learned from  Hurricane Katrina, the White House report made over 100  recommendations and concluded that an immediate priority for correcting  the shortfalls in the federal response to Hurricane Katrina was to define  and implement a comprehensive national preparedness system.  We  believe a strategic plan for implementing the national preparedness system  that includes a clearly stated purpose, scope, and methodology could help  the Preparedness Directorate convey to preparedness stakeholders the  importance of integrating the multiple elements of the national  preparedness system and interagency coordination.
Problem definition and risk assessment: As shown by 9/11 and  Hurricane Katrina, the nation faces risks from terrorist attacks and man- made and natural disasters.  These threats may vary between localities and  regions, but responders are to be able to effectively work together in a  common language on operational tasks when required.  According to the  National Preparedness Guidelines, responders must identify and assess  risk to ensure the necessary capabilities are available for selecting the  appropriate response.  In addition, understanding risk involves assessing  what vulnerabilities and weaknesses require further attention.  The lessons  learned from Hurricane Katrina show that federal agencies were not  prepared for a catastrophic disaster.  Confusion by emergency responders  over their roles and responsibilities was widespread and resulted in a slow  or fragmented response.  To improve response and recovery to all hazards  including a catastrophic disaster, the Post-Katrina Act called on FEMA to  develop the national preparedness system.
Goals, subordinate objectives, activities, and performance  measures: The cultural shift of the preparedness community from a  response and recovery strategy to a proactive preparedness strategy  emphasizes the importance of a strategic plan that includes clear goals,  objectives, activities, and measures.  Identifying performance measures for  the various components of the national preparedness system, which is also  a requirement under the Post-Katrina Act, will help policymakers  determine what progress has been made and what remains to be done,  especially as it relates to preparedness for a catastrophic disaster.  A  strategic plan that outlines an overarching goal, subordinate objectives,  activities, and performance measures for the various components of the  national preparedness system would help FEMA prioritize future efforts  and allow decision makers to measure progress.
Resources, investments, and risk management: Preparedness  agencies are to manage their likely risks and direct finite resources to the  most urgent needs.  The national preparedness system helps inform  decision makers in federal and state agencies on their use of resources  relative to their level of capabilities achievement.  Different states and  areas face different risks, and thus should have different capabilities to  mitigate those risks.  As we reported in March 2008, although DHS has  taken some steps to establish goals, gather information, and measure  progress, its monitoring of homeland security grant expenditures does not  provide a means to measure the achievement of desired program  outcomes. FEMA’s current efforts do not provide information on the  effectiveness of those funds in improving the nation’s capabilities or  reducing risk.  The National Strategy for Homeland Security describes  how resources and risk management must be addressed in a  comprehensive approach.  For example, the strategy states that “We  must apply a risk-based framework across all homeland security efforts in  order to identify and assess potential hazards, determine what levels of  relative risk are acceptable, and prioritize and allocate resources among  all homeland security partners, both public and private, to prevent, protect  against, and respond to and recover from all manner of incidents.”  A  strategic plan that outlines resources, investments, and risk management  would help FEMA coordinate a prioritized approach.
Organizational roles, responsibilities, and coordination: Achieving  national preparedness, especially for catastrophic incidents, requires  sharing responsibility horizontally with other federal departments and  agencies.  It also requires a robust vertical integration of the federal, state,  local, and tribal governments, as well as private entities.  FEMA’s  Preparedness Directorate faces the challenge of aligning operations of the  nation’s preparedness stakeholders to coordinate activities and plans to  implement a national preparedness program capable of dealing with  catastrophic incidents.  A key part of a national preparedness strategic  plan would be the clear delineation of organizations and their roles and  responsibilities, as well as processes to coordinate their responsibilities.
Integration and implementation: A national preparedness strategic  plan would help describe how preparedness agencies at all government  levels and sectors will integrate their various standards, policies, and  procedures into the national preparedness system.  Plans describing how  to integrate and implement the various elements of the national  preparedness system would help FEMA inform emergency managers, first  responders, and decision makers on how the individual elements of the  national preparedness system will improve capabilities, training, and plans  for all hazards, including catastrophic disasters.  A strategic plan to  implement the national preparedness system would enable FEMA’s  Preparedness Directorate to improve its likelihood of achieving its vision,  evaluating progress, and ensuring accountability of federal agencies and  other organizations in aligning their efforts to develop and improve the  national preparedness system.  While it may be impossible to have  absolute compatibility because of the many public and private  organizations involved, the danger in organizations using different  methods or systems, without some overall guidance to assure consistent  application of approaches, is that the elements of the national  preparedness system will have little ability to inform one another.  More  importantly, these systems may produce unreliable or incomplete data on  how to improve programs related to response and recovery.  FEMA plays a  crucial role in this regard through its statutory responsibility of carrying  out the Post-Katrina Act requirements for the national preparedness  system.
section_title: Conclusions
paragraphs: The nation looks to FEMA for leadership to ensure that stakeholders  involved in preparedness activities can effectively provide a coordinated  response to man-made or natural disasters.  The nation’s experiences after  the events of 9/11 and the 2005 hurricane season dramatically  demonstrated our emergency preparedness capabilities and where they  were lacking.  The Post-Katrina Act’s centralization of responsibility in  FEMA for exercises and the other primary activities that form the national  preparedness system provides an unprecedented opportunity for  comprehensive integration and coordination.  While FEMA has made  progress in implementing each of these interdependent and essential  preparedness activities, it is difficult to measure this progress.  FEMA  lacks a comprehensive approach to managing the development of policies  and plans and overseeing the National Exercise Program.  Additionally,  FEMA has not established a clearly defined course of action to assess  capabilities based on quantifiable metrics.  Finally, FEMA has not  established a strategic plan for integrating these elements of the national  preparedness system.  These conditions show that much remains to be  done.  In the short term, progress is heavily dependent on continuing to  improve basic policies and procedures, management tools, and project  plans for key elements of the system.  Developing each element of the  system is undoubtedly a complex task, but progress has to be built on  these incremental but critical steps.  In the long term, progress will be  increasingly dependent on how well FEMA coordinates with the  thousands of stakeholders in the system and the degree to which it can  integrate the plans, exercises, and assessments into a cohesive approach  that improves national preparedness.  This need centers attention on  leadership and guidance from the Preparedness Directorate and success  will depend on linking the various elements of the system and showing  how data and information from the system will inform program and  budget decisions related to improving preparedness.
A complete, integrated set of national preparedness policies and plans that  defines stakeholders’ roles and responsibilities at all levels is needed to  ensure that federal, state, and local resources are invested in the most  effective exercises.  As one program official told us, “If an exercise is  testing an inadequate plan, then the exercise is just an experiment….”   Until all national preparedness policies are developed and operational  plans are created or revised to reflect changes in the roles and  responsibilities of key stakeholders, FEMA’s ability to update requisite  training to prepare officials responsible for fulfilling these roles and reflect  the preparedness lessons of the unprecedented disasters of the last decade  in exercises and real-world response will be limited.  Without a program  management plan, FEMA cannot effectively ensure, in coordination with  DHS and other federal entities, that it will complete and integrate key  policies and plans with each other and the national preparedness system  as envisioned by law and presidential directive.
In implementing the latest iteration of the National Exercise Program,  FEMA has issued guidance and requirements for exercise design,  execution, evaluation, and corrective action resolution.  However, federal  and state exercise officials have not yet fully embraced the essential  program components.  In addition, opportunities to make exercises as  realistic as possible by coordinating more fully with all preparedness  stakeholders, including the National Council on Disability, and translating  the experiences of real-world incidents into corrective actions could  further enhance the value of the exercise program.  Establishing policies  and procedures that detail how FEMA would work with federal entities as  well as monitor states’ compliance in implementing the program would  help stakeholders meet program requirements and FEMA develop  complete and accurate information on program implementation.  More  importantly, these key program controls, once more systematically  established and applied, will enhance FEMA’s ability to assess the extent  to which corrective actions have been implemented and, ultimately,  describe strengths and weaknesses in the nation’s preparedness  capabilities.
Because a comprehensive assessment of national preparedness  capabilities is a monumental task, it is understandable that FEMA’s efforts  to develop and implement an assessment approach have been underway  for more than a decade.  Program officials with responsibility for this most  recent effort have recognized the need for a comprehensive set of metrics  to identify needed capabilities in equipment, personnel, skills, or  processes and to prioritize national investments in preparedness.  Given  the complexity of this effort, they would benefit from a clear roadmap that  details their analytical approach for integrating disparate information  sources, identifies associated program risks, establishes more specific  milestones to help avoid unexpected setbacks,  and provides a basis for  assessing program progress and making revisions, if needed, to the  agency’s implementation plans.  Such mitigation efforts would not  eliminate the risks associated with the development of the comprehensive  assessment system and target capabilities metrics, but they would provide  a basis for holding officials responsible for timely and quality results and  hold them harmless for unavoidable or unforeseen events that could delay  their efforts.
Finally, effective coordination, integration, and implementation of these  elements of the national preparedness system require the combined  contributions of a broad range of federal, state, and local stakeholders.   FEMA has started this integration effort and has had some success in  issuing guidelines and requirements that seek greater uniformity of effort.   But our work shows that issuing guidelines alone does not assure  consistent application across organizations.  While it may be impossible to  have absolute compatibility because of the many public and private  organizations involved in preparedness, the danger in organizations using  different methods or systems—without some overall guidance, direction,  and controls in place to assure consistent application of preparedness  approaches—is that the elements of the preparedness system will have  little ability to inform one another.  Perhaps, more important, these  systems may produce unreliable or incomplete data on how to allocate  resources or to improve programs related to response and recovery,  especially with respect to catastrophic incidents.  In this regard, FEMA  officials have noted that their authority is limited to coordinating with, but  not directing, other federal agencies.  This condition highlights the  importance of developing a strategic approach that leads to partnerships  with stakeholders whose cooperation is necessary for developing the  preparedness system.  Making progress with regard to this challenge is a  necessary step to assessing our nation’s capabilities and dealing with gaps  in preparedness.  The scope and breadth of this critical national effort  suggests that an explicit description and elaboration of the elements of the  system and the level of effort associated with its effective application  could enhance stakeholder acceptance and participation.  In addition,  defining the end state of the preparedness system will help translate  requirements from presidential directives and the Post-Katrina Act into  measurable steps for achieving an integrated national system.  Developing  goals and metrics to measure progress towards achieving an integrated  system will help FEMA prioritize actions, requirements, and national  investments in preparedness.  A strategic plan for the National  Preparedness Directorate that describes how it will approach these  challenges and mitigate these weaknesses would help FEMA partner with  the many organizations whose cooperation and resources are necessary  for success.
section_title: Recommendations for Executive Action
paragraphs: To ensure that key elements of the national preparedness system are  developed in a timely and integrated fashion, we recommend that the  Administrator of the Federal Emergency Management Agency take the  following 11 actions:     Direct the Disaster Operations Directorate and the National Preparedness  Directorate to improve their approach to developing policies and plans  that define roles and responsibilities and planning processes.
Develop a program management plan, in coordination with DHS and  other federal entities, to ensure the completion of the key national  preparedness policies and plans called for in legislation, presidential  directives, and existing policy and doctrine, to define roles and  responsibilities and planning processes, as well as to fully integrate  such policies and plans into other elements of the national  preparedness system.  The program management plan, among other  things, should:  identify the specific schedule of activities that needs to be performed  to complete, and identify dependencies among, all policy and planning  development and integration activities;  identify the type and quantities of resources required to perform, and  the schedule for completing, all policy and planning development and  integration activities;     analyze activity sequences, durations (including the time required to  partner and coordinate on an interagency basis with other federal  entities), resource requirements, and schedule constraints to create  and update the individual policy and plan development project  schedules; and     control for changes to the project schedules precipitated by outside  forces.
When outstanding policies and plans are completed, integrate them into  training and exercise efforts to ensure that roles and responsibilities are  fully communicated and fully understood by emergency response  stakeholders.
Direct the National Exercise Division to improve its implementation of  statutory and program requirements.
Coordinate with the Department of Homeland Security to develop  policies and procedures for issuing after-action reports for National  Level Exercises (i.e., TOPOFF) in 6 months or less, as required by the  implementation plan for the National Exercise Program.
Collaborate with the Homeland Security Council to establish policies  and procedures for documenting corrective actions from Principal  Level Exercises that are consistent with HSEEP guidance and the  implementation plan for the National Exercise Program.
Collaborate with the Homeland Security Council to provide FEMA with  the information it needs from past principal level exercises to enable it  to conduct remedial action tracking and long-term trend analysis, as  required by the Post-Katrina Act.
Ensure compliance by states that receive grant funds with HSEEP  requirements by revising FEMA’s grant monitoring guidance, for  example by including a checklist of specific HSEEP requirements for  state validation and certification.
Involve the National Council on Disability on committees involved in  the design and execution of national level exercises, especially on  issues related to populations with special needs.
Develop internal control policies and procedures that validate the  completeness and accuracy of data used to measure program  performance.  Such procedures could involve checking whether states  and federal agencies are providing data and information needed to  measure the performance of the program.
Revise the National Exercise Program Implementation Plan to require  the use of FEMA’s Corrective Action Program for all federal exercises  that involve interagency testing of roles and responsibilities or require  that federal agencies submit a report to FEMA on the status of their  corrective actions resulting from such exercises.
Develop procedures for including “lessons learned” from real-world  incidents in the Corrective Action Program system.
Direct the Office of Preparedness Policy, Planning, and Analysis to  improve its approach for developing a comprehensive assessment system.
Enhance its project management plan to include milestone dates, an  assessment of risk, and related mitigation strategies for (1)  comprehensively collecting and reporting on disparate information  sources, (2) developing quantifiable metrics for target capabilities that  are to be used to collect and report preparedness information, and (3)  reporting on the results of preparedness assessments to help inform  homeland security resource allocation decisions.
Direct the National Preparedness Directorate to take a more strategic  approach to developing the national preparedness system.
Develop a strategic plan for implementing the national preparedness  system that includes the key characteristics of a strategic plan,  including coordination, integration, and implementation approaches.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for comment.  In commenting on  our draft report, DHS noted that while it may not agree with all the  assertions in the report, it generally agreed with our recommendations.   DHS stated that GAO’s recommendations provide a useful methodology  and sound counsel for revision of FEMA’s current portfolio of national  preparedness policy, plans, protocols, and procedures. Specifically, DHS  stated that FEMA has already made significant inroads in each aspect of  the GAO recommended characteristics for sound strategic planning.
DHS also expressed concern that the report suggests that DHS/FEMA  should hold other federal agencies and departments or state, local, or  tribal governments accountable for compliance with program  requirements, while also recognizing that FEMA did not generally have the  explicit authority to compel compliance.  The Post-Katrina Act designates  FEMA as the federal leader and coordinator for developing and  implementing the national preparedness system. We recognize that  FEMA’s authority is generally to coordinate, guide, and support, rather  than direct, and that collaboration is an essential element of FEMA’s  efforts.  At the same time, we believe that FEMA’s expanded leadership  role under the Post-Katrina Act provides FEMA opportunities for and a  responsibility to further develop its relationships with national  preparedness stakeholders at the local, state, and federal levels and to  instill a shared sense of responsibility and accountability on the part of all  stakeholders for the successful development and implementation of the  national preparedness system.  Several of our recommendations aim to  enhance such collaboration and cooperation.
DHS also provided technical comments, which we incorporated into the  report as appropriate.  Appendix V contains written comments from DHS.
As agreed with your office, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from the  date of this report.  At that time we will send copies of this report to the  Secretary of Homeland Security, the Director of the Office of Management  and Budget, and interested congressional committees.  We will also make  copies available to others on request.  In addition, this report will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-8757 or e-mail at jenkinswo@gao.gov.  Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report.  Key contributors to this report are  listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This review examined key elements of the national preparedness system,  including the National Exercise Program.  Specifically, our reporting  objectives were to review the extent to which the Federal Emergency  Management Agency (FEMA) has:   1.  developed policies and plans that define roles and responsibilities and  planning processes for national preparedness;  2.  taken actions since 2007 to implement the National Exercise Program  and track corrective actions at the federal and state levels and what  challenges remain;  3.  made progress in conducting a nationwide capabilities-based  assessment, including developing required preparedness reports, and  what issues, if any, it faces in completing the system; and,   4.  developed a strategic plan for implementing the national preparedness  system.
To address these objectives, we analyzed information and data on FEMA’s  policies and plans for preparedness, the National Exercise Program, its  approach for developing a comprehensive system for assessing nationwide  capabilities, and its strategy for integrating elements of the preparedness  system.  GAO explored the option of selecting exercises to review, but  there is no national database that captures all exercises conducted using  Homeland Security Grant Program funds.  Therefore we selected six  states—California, Georgia, Illinois, New York, Texas, and Washington— that provide examples of how exercises are planned and conducted and  visited these six states.  While we cannot generalize our work from these  visits to all states, we chose these locations to provide examples of the  way in which states carry out their exercise and preparedness programs.   In selecting these states, we considered factors such as states’  participation in national-level exercises; states located in different  geographic locations, such as those in hurricane-prone regions; and states  with varying percentages of homeland security grant funding planned to  support exercises. At each location, we interviewed staff in FEMA’s  regional offices responsible for regional preparedness activities. We  interviewed state and local officials on their progress and challenges in  carrying out preparedness activities, including exercises and assessments  of capability.  We analyzed key legislation such as the Post-Katrina Act and  the Implementing Recommendations of the 9/11 Commission Act of 2007  (9/11 Act) as well as presidential directives related to preparedness efforts.
We also interviewed FEMA officials responsible for preparedness  programs to learn more about the actions they had taken and planned to  take related to preparedness efforts and compared FEMA’s policies and  procedures with criteria in GAO’s standards for internal control in the  federal government.
To analyze the extent to which policies and plans have been developed to  define roles and responsibilities and planning processes for national  preparedness, we analyzed key legislation, presidential directives, and  DHS- and FEMA-issued policies that identify required preparedness  policies and plans to define roles and responsibilities for emergency  response as well as establish guidance for planning processes for  developing emergency response plans.  We identified the resulting policies  that define roles and responsibilities and that form the basis of the  national preparedness system, including the National Strategy for  Homeland Security, the National Response Framework (NRF), the  National Preparedness Guidelines, and the National Incident Management  System.  In addition, we identified related policies that supplement these  documents, such as guidance for Joint Field Office operations.  We also  identified policies that define planning processes for developing  emergency plans, such as the draft Integrated Planning System (IPS) and  FEMA-issued interim and final Comprehensive Preparedness Guides for  nonfederal planning efforts.  We identified plans developed using planning  processes that further define and operationalize roles and responsibilities  identified in existing policies.  These plans include the incident annexes  and the incident annex supplements for the NRF, FEMA’s Pre-Scripted  Mission Assignment Catalog, plans being developed as part of FEMA’s  Catastrophic Disaster Planning Initiative, as well as plans called for by  HSPD 8 Annex 1 that are to be developed using IPS.  To identify lessons  learned and corrective actions related to roles and responsibilities from  federal emergency response exercises, we summarized lessons learned  from after-action reports for Tier I and II (or equivalent) exercises from  2005 through 2008.  The exercises that comprised this data set were  identified by FEMA officials as well as counsel to the White House  Homeland Security Council.  This analysis was conducted to determine  whether the exercises revealed unclear or conflicting roles and  responsibilities between federal departments and agencies and if  additional policies and plans were needed.  We also interviewed officials  from DHS’s Office of Operations Coordination and FEMA’s National  Preparedness Directorate and Disaster Operations Directorate to obtain  information on the status of efforts to develop and issue required  preparedness policies and plans, including any existing program or project  management plans and related issuance schedules.  We compared policies  and plans that have been published in a final form, versus released in  interim or draft formats or that have not yet been developed, to determine  the issuance status (completed, partially completed, or incomplete) of  these policies and plans.  To identify best practices for program  management, such as steps for how a program is to be executed,  monitored, and controlled, we reviewed the Project Management  Institute’s The Standard on Program Management.  Finally, we reviewed  our prior reports on FEMA’s preparedness programs and planning efforts,  as well as prior DHS, White House, and congressional reports on the  lessons learned from the response to Hurricane Katrina in 2005.
To assess the extent to which FEMA has taken actions since 2007 to  implement a National Exercise Program and track corrective actions, we  observed portions of two National Exercise Program exercises (TOPOFF 4  and the May 2008 Tier II exercise).  Specifically, during TOPOFF 4, we  discussed exercise implementation with federal, state, and local officials  and observed FEMA’s exercise management efforts at the TOPOFF 4  Master Control Cell in Springfield, Virginia; the TOPOFF 4 Long-Term  Recovery Tabletop Exercise in Washington, D.C.; and exercise  implementation in Portland, Oregon.  During the May 2008 Tier II exercise,  we observed exercise implementation in Mount Weather and Suffolk,  Virginia and in Blaine, Washington and discussed the exercise with  participating federal, state, and local officials.  We evaluated key program  documents, such as the implementation plan for the National Exercise  Program and the Homeland Security Exercise and Evaluation Program  (HSEEP)—FEMA’s guidance for carrying out exercises in accordance with  the Post-Katrina Act and the 9/11 Act and data on the program’s  performance measures.  We reviewed actions taken by FEMA since 2007  because the National Exercise Program Charter was established in  January 2007 and the implementation plan was issued in April 2007.  We  examined after-action reports for Principal Level Exercises-–exercises  which involve senior federal officials, such as Deputy Secretaries of  departments or agencies-–that were issued from April 2007 through August  2008 to determine whether the Homeland Security Council developed  after-action reports.  We also interviewed Homeland Security Council  staff and the Associate Counsel to the President on the role and  responsibility of the council for systematically tracking and implementing  corrective actions resulting from Principal Level Exercises because these  staff were responsible for summarizing corrective actions for Principal  Level Exercises.  We also reviewed after-action reports that were provided  to us by the six states we visited for exercises conducted from June 2007  through September 2008 that used Homeland Security Grant Program  funds, in order to determine how well these states were complying with  HSEEP and grant guidance.  To determine if FEMA is conducting  monitoring and oversight of Homeland Security Grant Program recipients,  we reviewed grant monitoring reports for the six states we visited.  For  information that would provide a broader perspective on FEMA’s efforts,  we examined several FEMA databases, including the FEMA Secure  Portal—the FEMA repository of after-action reports; the National Exercise  Schedule (NEXS) system—a scheduling system for all exercises; and the  Corrective Action Program (CAP) system—which is designed for tracking  capability-based improvement plans entered by federal, state, and local  exercise participants. We assessed the reliability of the FEMA Secure  Portal and NEXS databases by checking the systems to determine if  known exercises identified through after-action reports produced by  states were included in these systems, and by interviewing FEMA and  state officials responsible for the data.  In addition, we assessed the  reliability of the CAP system by interviewing FEMA and Homeland  Security Council officials responsible for the data.  We concluded the data  in the FEMA Secure Portal, the NEXS system, and the CAP system were  not reliable for use in this report because these databases lacked complete  information related to after-action reports, scheduled exercises, and  corrective actions and FEMA does not have procedures in place to ensure  that required data are collected consistently to populate these databases.
To determine the extent to which FEMA has made progress and issues  FEMA has encountered in conducting a nationwide capabilities-based  assessment and developing required preparedness reports and any issues  it faces in completing the system, we (1) analyzed FEMA’s plans and  schedules for developing the comprehensive assessment system and  performance objectives for measuring capabilities, including assessment  efforts initiated by FEMA’s National Preparedness, Disaster Operations,  and Grant Programs Directorates, and (2) interviewed FEMA staff  responsible for these efforts.  We also reviewed assessments previously  conducted by DHS and FEMA to evaluate historical efforts to assess  capabilities.  To assess FEMA’s efforts to establish quantifiable metrics for  target capabilities, we analyzed preliminary performance objectives for  two target capabilities that FEMA had completed, and interviewed  headquarters staff responsible for these efforts.  We also reviewed  information pertinent to FEMA’s assessment approach, including the  Federal Preparedness Report (issued by FEMA in January 2009) and State  Preparedness Reports for the 2007 reporting year for the six states we  visited (reports for 2008 were due to FEMA after this report was finalized  for publication).  To assess the comparability of information contained in  the six State Preparedness Reports, we selected one target capability— mass prophylaxis—outlined in the Target Capabilities List, and reviewed  a performance measure associated with this capability for providing initial  prophylaxis (an action taken to prevent a disease or a health problem)  within 48 hours of a state/local decision to provide prophylaxis.  To  identify best practices for project management, such as steps for how a  project is to be executed, monitored, and controlled, we reviewed the  Project Management Institute’s A Guide to the Project Management Body  of Knowledge (PMBOK) and compared FEMA’s efforts for developing and  implementing the comprehensive assessment system to the best practices  developed by the institute.  Finally, we reviewed our prior GAO reports on  FEMA’s preparedness programs and exercise efforts.
To determine the extent to which FEMA’s National Preparedness  Directorate has developed a strategic plan that implements the national  preparedness system, we interviewed FEMA National Preparedness  Directorate officials on strategic planning and policy and procedures for  the National Preparedness System.  We analyzed the Post-Katrina Act and  key FEMA documents—including the agencywide Strategic Plan for Fiscal  Years 2008 through 2013, the Grants Program Directorate Strategic Plan,  and the draft annual National Preparedness Directorate Operating Plan— to determine strategy-related requirements.  Additionally we reviewed  DHS and DHS Office of Inspector General reports on national  preparedness.  To determine the elements that comprise a strategic plan,  we examined our prior reports on the desirable characteristics of effective  national strategies and compared them with FEMA’s current approach for  developing a National Preparedness System.  Finally, we compared the  desirable characteristics to the work of the National Preparedness  Directorate as described in the Homeland Security Council’s National  Strategy for Homeland Security, DHS’s National Preparedness Guidelines,  and the Post-Katrina Act.
We conducted this performance audit from January 2008 through April   2009 in accordance with generally accepted government auditing  standards.  Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives.  We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Overview of Policies and Plans to Define Roles and Responsibilities and Planning Processes for Developing Emergency Plans
paragraphs: In September 2006, we reported that in preparing for, responding to, and  recovering from any disaster, the legal authorities, roles and  responsibilities, and lines of authority for the preparation and response at  all levels for government and nongovernmental entities must be clearly  defined, effectively communicated, and well understood in order to  facilitate rapid and effective decision making.  National preparedness  policies and plans identify these legal authorities, roles and  responsibilities, and lines of authority for response activities and serve to  communicate this information to emergency response stakeholders, and,  in conjunction with training, are the basis for ensuring that the  information is well understood.  Effective and efficient disaster  management relies on the thorough integration of these emergency  response policies and plans.  Because emergency response activities entail  large numbers of stakeholders who need to be able to respond to an  incident in a coordinated and integrated manner, it is essential that roles  and responsibilities are defined, communicated, and understood prior to a  real-world incident response.  An example of the range of stakeholders  involved in such response activities is illustrated by figure 6.
The wide range of emergency response stakeholders depicted in figure 6,  among others, are to be organized by the roles and responsibilities defined  in policies and plans that are designed to facilitate an effective response to  an incident, be it man-made (e.g., terrorism) or a natural disaster.  Policies  that broadly define roles and responsibilities are operationalized by the  development of plans that provide greater levels of detail.  These detailed  plans are to be developed using the planning processes that are discussed  and established in federal policies.  The range and relative relationships of  the policies that define roles and responsibilities and planning processes  for developing emergency plans are illustrated in figure 7.  Table 4, which  follows figure 7, describes each of the policies presented in the graphic.
Among the 50 policies that define roles and responsibilities or planning  processes, 46 define roles and responsibilities and 4 define planning  processes for developing emergency plans.  Table 4 provides brief  descriptions and the status of the policies depicted in figure 7 that define  roles and responsibilities.  Of the 46 policies presented in table 4, 40 have  been completed and 6 are incomplete. a standardized federal planning process (the Integrated  Planning System);   resourced operational and tactical planning capabilities at  each federal department and agency with a role in homeland  security;   strategic guidance statements, strategic plans, concepts of  operations, operations plans, and, as appropriate, tactical  plans for each National Planning Scenario; and   a system for integrating plans among all levels of  government.
It also calls for the development of the National Homeland Security  Plan.
Policy and doctrine (Issued by DHS or FEMA)
The NSHS is to guide, organize, and unify homeland security efforts  by providing a common framework for the prevention of terrorist  attacks; protection of people, critical infrastructure, and key resources;  and response to and recovery from man-made and natural disasters.   It calls for homeland security management through a continuous,  mutually reinforcing cycle of four activity phases:   (July 2002; revised  October 2007)
1.  overarching homeland security guidance grounded in clearly  articulated and up-to-date homeland and relevant national  security policies, with coordinated supporting strategies,  doctrine, and planning guidance flowing from and fully  synchronizing with these policies;   2.  a deliberate and dynamic system that translates policies,  strategies, doctrine, and planning guidance into a family of  strategic, operational, and tactical plans;  3. the execution of operational and tactical-level plans; and   4.  continual assessment and evaluation of both operations and  exercises.
NHSP, as called for under HSPD 8 Annex 1, is to be an overarching  strategic plan to guide national efforts to execute the National Strategy  for Homeland Security. It is intended to:  facilitate federal homeland security coordination,  (release schedule  undetermined) define roles and responsibilities for preventing, protecting  against, responding to, and recovering from man-made and  natural disasters.
It was to be submitted to the President for approval within 120 days of  the approval of HSPD 8 Annex I (December 2007); however, as of  April 2009, the NHSP has yet to be published.
NIMS presents a core set of doctrine, concepts, principles,  procedures, organizational processes, terminology, and standard  requirements designed to enable effective, efficient, and collaborative  incident management. It forms the basis for interoperability and  compatibility to enable a diverse set of public and private organizations  to conduct integrated emergency management and incident response  operations. (March 2004; revised  December 2008)  (interim NPG, March  2005; final NPG and 3  capabilities-based  preparedness tools,  September 2007)
The NPG consists of a vision, capabilities, and priorities for national  preparedness. It establishes three capabilities-based preparedness  tools (Universal Task List, Target Capabilities List, and National  Planning Scenarios) and the National Preparedness System cycle  (plan; organize and staff; equip; train; and exercise, evaluate, and  improve) to collate existing homeland security plans, strategies, and  systems into an overarching framework. The revised and finalized  NPG replaces the interim National Preparedness Goal released in  March 2005.
The NRF is a guide to how the nation conducts all-hazards response,  generally describing national response doctrine and the roles and  responsibilities of officials involved in response efforts, including,  among others, the Secretary of Homeland Security, FEMA  Administrator, Principal Federal Official, and Federal Coordinating  Officer. It is designed to align key roles and responsibilities, linking all  levels of government, nongovernmental organizations, and the private  sector, as well as capture specific authorities and best practices for  managing incidents that range from the serious but purely local, to  large-scale terrorist attacks or catastrophic natural disasters. (January 2008)
NRF ESF Annexes align categories of federal government response  resources and capabilities and provide strategic objectives for their  use under the NRF. They provide the structure for coordinating federal  interagency support for a federal response to an incident and are  mechanisms for grouping functions most frequently used to provide  federal support to states, and federal-to-federal support. Each ESF  Annex, such as Search and Rescue, identifies the federal agency  coordinator and the primary and support agencies pertinent for the  ESF, and when activated, the initial actions delineated in the ESF  Annexes guide response activities. (January 2008)
NRF Support Annexes describe the roles and responsibilities of  federal departments and agencies, nongovernmental organizations,  and the private sector in coordinating and executing the functional  processes and administrative requirements necessary for incident  management that are common to all incidents. They identify the  federal agency coordinator and the primary and support agencies  pertinent for the support activity, such as financial management or  private-sector coordination, and when activated, the initial actions  delineated in the Support Annexes guide response activities. (January 2008)
NRF Partner Guides are to provide stakeholder-specific references  describing key roles and actions for local, tribal, state, federal, private- sector, and nongovernmental response partners. They are to  summarize core NRF concepts and be tailored specifically to leaders  at different levels of government and from different types of  organizations.  As of April 2009, none of the four NRF Partner Guides  have been published. (release schedule  undetermined)
Completed   (April 2006)
Completed   (April 2006)
The various Joint Field Office (JFO) guides were written to support the  December 2004 National Response Plan for the establishment of  JFOs.  The JFO is a temporary federal multi-agency coordination  center established locally to facilitate coordinated field-level domestic  incident management activities, with the JFO IISOP, JFO A&A, and  JFO FOG designed to assist personnel assigned to response  operations.  In particular, the JFO IISOP and JFO A&A provide  detailed guidance on JFO activations and operations, including  defining the roles and responsibilities and concept of operations for the  Principal Federal Official cell within the JFO.  The JFO FOG is  intended to be used as a quick reference job aid for JFO personnel.
Completed   (June 2006)
According to FEMA officials, FEMA’s Disaster Operations Directorate  is developing a JFO Organization & Functions Manual, reflecting the  NRF, as an addition to existing JFO guidance.  This new manual is to  provide guidance for the establishment, operation, and demobilization,  as well as the general organization and staffing, of JFOs  (release schedule  undetermined)
The type column identifies the categorization of policy and plans.  Policies are categorized under the  term “policy & doctrine” (including legislation, presidential directives, and other policies that define  roles and responsibilities, as well as policies that define planning processes), per the national  preparedness cycle presented in the National Preparedness Guidelines.  Plans that define roles and  responsibilities and that are to operationalize policies are categorized under the term “planning &  resource allocation,” also per the national preparedness cycle presented in the National  Preparedness Guidelines.
Table 5 provides brief descriptions and the status of the policies that  define planning processes for developing the emergency plans depicted in  figure 7.  Of the four policies presented in the table, two have been  completed and two are partially completed.
The range and relative relationships of the plans that define roles and  responsibilities are illustrated in figure 8.  Table 6, which follows figure 8,  describes each of the plans presented in the graphic.
Table 6 provides brief descriptions and the status of the plans that define  roles and responsibilities that are depicted in figure 8.  Of the 72 plans  presented in the table, 20 have been completed, 3 have been partially  completed, and 49 are incomplete.
As noted in table 6, a schedule exists for the release of the myriad plans  called for under HSPD 8 Annex 1 using the not yet published Integrated  Planning System (IPS).  Figure 9 presents the schedule for the release of  specific plans under IPS.
Finally, figure 10  shows the combined universe of all the policies and  plans in figures 7 and 8 in relation to each other.  Figure 11, which follows,  shows the status of development of each of the policies and plans.
section_title: Appendix III: Information Regarding Assessments Used to Develop the Comprehensive Assessment System
paragraphs: This appendix presents additional information regarding the Federal  Emergency Management Agency’s (FEMA) progress and any remaining  issues it faces in conducting a nationwide comprehensive assessment  system.  FEMA has identified various ongoing and historical assessment  efforts that it plans to use to inform the development of the  comprehensive assessment system.  Additional information regarding  these efforts is outlined below.
State Preparedness Reports. While state preparedness reports  assess capabilities within states, FEMA could not use information in  the 2007 reports to compare capability gaps between states, because  states did not report information using common metrics to assess  capabilities and data were not always available to consistently  complete the report.  FEMA’s state preparedness guidance explains  that states are to “use relevant metrics . . . from the Target Capabilities  List when describing current capabilities.”  However, FEMA has not  developed a framework for states to use in reporting their current  capabilities against the target capabilities because FEMA is in the  process of (1) developing quantifiable metrics for the target capabilities  and (2) revising the reporting format for state preparedness reports in  order to base them on the target capabilities.  As a result, the 2007  reports do not report state capabilities in a measurable way, or with the  level of detail necessary for a comparison across states and territories.   In addition, the six states we visited used different techniques to  summarize their capabilities.  In one location, a state homeland  security task force held discussion groups to determine what the  capability needs are and what resources are needed.  In another  location, the state held a workshop attended by stakeholders from  across the state to collect and obtain input on capability needs to  complete the state preparedness report.  Two states relied on  information collected for their respective state homeland security  strategies.  A fifth state primarily used information it had gathered to  prepare a grant funding reporting requirement, while officials at a sixth  state collected information through site visits.  FEMA headquarters  officials explained that they intend to use the target capabilities as the  framework for future state preparedness reports.
National Incident Management System Compliance Assessment  Support Tool (NIMSCAST). NIMSCAST is a Web-based tool to  assess states’ and territories’ compliance with the NIMS, which is a  standardized process by which emergency responders are to conduct  integrated incident response operations, rather than a method for  assessing capabilities.  Assessing compliance with National Incident  Management System requirements is one of the requirements for the  comprehensive assessment system.  In February 2009, FEMA  indicated that it will use NIMSCAST to continue collecting data on  NIMS compliance in addition to collecting capability and state  preparedness report data through a survey it will distribute in 2009.   FEMA noted that this effort will consolidate reporting requirement s  and fulfill Post-Katrina Act requirements for the comprehensive  assessment system.  In addition, FEMA noted that it will use d collected through NIMSCAST related to compliance with incident  management processes and procedures to directly inform the of two target capabilities: On-Site Incident Management and  Emergency Operations Center Management.  However, it is unclear  how FEMA will integrate this tool with other features of the  comprehensive assessment system to assess the remaining 35  capabilities that will not be directly informed by NIMSCAST data  related to compliance with incident manag procedures.
FEMA and localities better target and measure the results of using  federal grant funds.  As its organizing structure, the C2C uses th National Planning Scenarios, such as a hurricane, earthquake,  improvised explosive device, or anthrax attack. Data for the C2C are t be based on self assessments of capabilities from state preparedne plans, estimates of baseline capability, and the estimated relative  capability improvement expected from a requested level of grant  investment.  To be used effectively and enable comparisons across  jurisdictions in evaluating grant proposals, the state and local data for  assessing state and local capabilities must be in the common language of target capabilities and have metrics that are compatible with C2C.   These metrics are being developed by FEMA’s National Preparedness  Directorate. However, grantee use of C2C will not be mandatory, a nd  thus its ultimate value is yet to be determined.  In developing the  initiative, grant officials are considering ways to collect data from  stakeholders with a minimal burden and integrate analyses re from the C2C into existing programs, plans, and procedures.
National Preparedness System (NPS).  The NPS was discontinuedta.  because it was time consuming and did not produce meaningful da This Web-based management information system was designed to  serve as an inventory tool to measure a jurisdiction’s ability to deliver  elements of planning, organization, equipment, training, and ex Although it was developed in response to Homeland Security  Presidential Directive 8’s preparedness requirements, in conjunction  with the Target Capabilities List, and pilot tested in 10 states, the  system was discontinued by the Department of Homeland Security  because officials said it was too time consuming to use, according to  FEMA officials.  Because it was only piloted, FEMA officials explained  that it did not generate meaningful preparedness information from the  data collected.  According to FEMA budget documentation required by  the Office of Management and Budget for major information  technology investments, FEMA spent nearly $15 million in total on the  system for 2006, 2007, and 2008 before it was discontinued.    ercises.
Pilot Capability Assessment (PCA).  The PCA was labor intensive  he  and did not generate meaningful data.  This assessment, based on t 37 target capabilities, was also intended to measure jurisdictions’  progress in achieving needed target capabilities.  While it was  developed in response to HSPD 8 preparedness requirements and in  conjunction with the Target Capabilities List and pilot tested in six  states, FEMA officials said it was too labor intensive.  Because only piloted, FEMA did not generate meaningful preparedness  information from the data collected, according to FEMA officials.
Capability Assessment for Readiness (CAR).  The CAR lacked  controls for validating the accuracy of self-reported assessment data.  This assessment was proposed as a one-time nationwide assessmen performance in areas such as planning and hazard management to  assess a national set of emergency management performance criteria  for FEMA grant recipients.  FEMA committed to preparing this  assessment in hearings before the Senate Committee on  Appropriations. The assessment was conducted in 1997 but concerns  reported by the DHS Inspector General in March 2006 regarding self- reporting and the lack of controls for validating information reported  by states limited the reliability and, therefore, the value of the data.   The assessment was conducted once in 1997.
section_title: Appendix IV: GAO’s Description of the Six Characteristics of an Effective National Strategy
paragraphs: In February 2004, we identified six desirable characteristics of an effective  national strategy that would enable its implementers to effectively shape  policies, programs, priorities, resource allocations, and standards and that  would enable federal departments and other stakeholders to achieve the  identified results. We further determined in that report that national  strategies with the six characteristics can provide policy makers and  implementing agencies with a planning tool that can help ensure  accountability and more effective results. To develop these six desirable  characteristics of an effective national strategy, we reviewed several  sources of information. First, we gathered statutory requirements  pertaining to national strategies, as well as legislative and executive  branch guidance. We also consulted the Government Performance and  Results Act of 1993, general literature on strategic planning and  performance, and guidance from the Office of Management and Budget on  the President’s Management Agenda. In addition, among other things, we  studied past reports and testimonies for findings and recommendations  pertaining to the desirable elements of a national strategy. Furthermore,  we consulted widely within GAO to obtain updated information on  strategic planning, integration across and between the government and its  partners, implementation, and other related subjects.
We developed these six desirable characteristics based on their underlying  support in legislative or executive guidance and the frequency with which  they were cited in other sources. We then grouped similar items together  in a logical sequence, from conception to implementation.  The following  sections provide more detail on the six desirable characteristics.
Purpose, scope, and methodology: This characteristic addresses why  the strategy was produced, the scope of its coverage, and the process by  which it was developed. For example, a strategy should discuss the  specific impetus that led to its being written (or updated), such as  statutory requirements, executive mandates, or other events like the global  war on terrorism. Furthermore, a strategy would enhance clarity by  including definitions of key, relevant terms. In addition to describing what  it is meant to do and the major functions, mission areas, or activities it  covers, a national strategy would ideally address its methodology. For  example, a strategy should discuss the principles or theories that guided  its development, the organizations or offices that drafted the document, or  working groups that were consulted in its development.
Problem definition and risk assessment: This characteristic addresses  the particular national problems and threats at which the strategy is  directed. Specifically, this means a detailed discussion or definition of the  problems the strategy intends to address, their causes, and operating  environment. In addition, this characteristic entails a risk assessment,  including an analysis of the threats to and vulnerabilities of critical assets  and operations. If the details of these analyses are classified or  preliminary, an unclassified version of the strategy should at least include  a broad description of the analyses and stress the importance of risk  assessment to implementing parties. A discussion of the quality of data  available regarding this characteristic, such as known constraints or  deficiencies, would also be useful.
Goals, subordinate objectives, activities, and performance  measures: This characteristic addresses what the national strategy strives  to achieve and the steps needed to garner those results, as well as the  priorities, milestones, and performance measures to gauge results. At the  highest level, this could be a description of an ideal end state, followed by  a logical hierarchy of major goals, subordinate objectives, and specific  activities to achieve results. In addition, it would be helpful if the strategy  discussed the importance of implementing parties’ efforts to establish  priorities, milestones, and performance measures that help ensure  accountability. Ideally, a national strategy would set clear desired results  and priorities, specific milestones, and outcome-related performance  measures while giving implementing parties flexibility to pursue and  achieve those results within a reasonable time frame. If significant  limitations on performance measures exist, other parts of the strategy  should address plans to obtain better data or measurements, such as  national standards or indicators of preparedness.
Resources, investments, and risk management: This characteristic  addresses what the strategy will cost, the sources and types of resources  and investments needed, and where those resources and investments  should be targeted. Ideally, a strategy would also identify appropriate  mechanisms to allocate resources. Furthermore, a national strategy should  elaborate on the risk assessment mentioned earlier and give guidance to  implementing parties to manage their resources and investments  accordingly. It should also address the difficult, but critical, issues about  who pays and how such efforts will be funded and sustained in the future.  Furthermore, a strategy should include a discussion of the type of  resources required, such as budgetary, human capital, information,  information technology (IT), research and development (R&D),  procurement of equipment, or contract services. A national strategy  should also discuss linkages to other resource documents, such as federal  agency budgets or human capital, IT, R&D, and acquisition strategies.  Finally, a national strategy should also discuss in greater detail how risk  management will aid implementing parties in prioritizing and allocating  resources, including how this approach will create society-wide benefits  and balance these with the cost to society. Related to this, a national  strategy should discuss the economic principle of risk-adjusted return on  resources.
Organizational roles, responsibilities, and coordination: This  characteristic addresses what organizations will implement the strategy,  their roles and responsibilities, and mechanisms for coordinating their  efforts. It helps to answer the question about who is in charge during times  of crisis and during all phases of national preparedness: prevention,  vulnerability reduction, and response and recovery. This characteristic  entails identifying the specific federal departments, agencies, or offices  involved, as well as the roles and responsibilities of private sectors. A  strategy would ideally clarify implementing organizations’ relationships in  terms of leading, supporting, and partnering. In addition, a strategy should  describe the organizations that will provide the overall framework for  accountability and oversight. Furthermore, a strategy should also identify  specific processes for coordination and collaboration between sectors and  organizations—and address how any conflicts would be resolved.
Integration and implementation: This characteristic addresses both  how a national strategy relates to other strategies’ goals, objectives, and  activities (horizontal integration)—and to subordinate levels of  government and other organizations and their plans to implement the  strategy (vertical integration). Similarly, related strategies should highlight  their common or shared goals, subordinate objectives, and activities. In  addition, a national strategy should address its relationship with relevant  documents from implementing organizations, such as the strategic plans,  annual performance plans, or the annual performance reports the  Government Performance and Results Act requires of federal agencies. A  strategy should also discuss, as appropriate, various strategies and plans  produced by the state, local, or private sectors. A strategy also should  provide guidance such as the development of national standards to link  together more effectively the roles, responsibilities, and capabilities of the  implementing parties.
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
paragraphs: William O. Jenkins Jr., (202) 512-8757 or jenkinswo@gao.gov.
subsections: section_title: Staff Acknowledgements
paragraphs: In addition to the contact named above, Chris Keisling (Assistant  Director), Neil Asaba (Analyst-in Charge), Joel Aldape, Avrum Ashery,   Tina Cheng, Brian Chung, Christine Davis, Lara Kaskie, Ron La Due Lake,  Brian Lipman, David Lysy, Jan Montgomery, and Robert Robinson made  key contributions to this report.
section_title: Related GAO Products
paragraphs: Actions Taken to Implement the Post-Katrina Emergency Management  Reform Act of 2006. GAO-09-59R.  Washington, D.C.: November 21, 2008.
Voluntary Organizations: FEMA Should More Fully Assess  Organization's Mass Care Capabilities and Update the Red Cross Role in  Catastrophic Events.  GAO-08-823. Washington, D.C.: September 18, 2008.
Emergency Preparedness: States Are Planning for Medical Surge, but  Could Benefit from Shared Guidance for Allocating Scarce Medical  Resources.  GAO-08-668. Washington, D.C.: June 13, 2008.
Emergency Management: Observations on DHS's Preparedness for  Catastrophic Disasters. GAO-08-868T.  Washington, D.C.: June 11, 2008.
National Response Framework: FEMA Needs Policies and Procedures to  Better Integrate Non-Federal Stakeholders in the Revision Process.  GAO-08-768. Washington, D.C.: June 11, 2008.
Homeland Security: DHS Improved its Risk-Based Grant Programs'  Allocation and Management Methods, But Measuring Programs' Impact  on National Capabilities Remains a Challenge. GAO-08-488T.  Washington, D.C.: March 11, 2008.
National Disaster Response: FEMA Should Take Action to Improve  Capacity and Coordination between Government and Voluntary Sectors.  GAO-08-369. Washington, D.C.: February 27, 2008.
Continuity of Operations: Selected Agencies Tested Various Capabilities  during 2006 Governmentwide Exercise. GAO-08-185. Washington, D.C.:  November 19, 2007.
Homeland Security: Observations on DHS and FEMA Efforts to Prepare  for and Respond to Major and Catastrophic Disasters and Address  Related Recommendations and Legislation.  GAO-07-1142T. Washington,  D.C.: July, 31, 2007.
Homeland Security: Preparing for and Responding to Disasters.  GAO-07-395T. Washington, D.C.: March 9, 2007.
Catastrophic Disasters: Enhanced Leadership, Capabilities, and  Accountability Controls Will Improve the Effectiveness of the Nation's  Preparedness, Response, and Recovery System.  GAO-06-618.   Washington, D.C.: September 6, 2006.
Emergency Preparedness and Response: Some Issues and Challenges  Associated with Major Emergency Incidents. GAO-06-467T. Washington,  D.C.: February 23, 2006.
GAO's Preliminary Observations Regarding Preparedness and Response  to Hurricanes Katrina and Rita. GAO-06-365R. Washington, D.C.:  February 1, 2006.
Homeland Security: DHS' Efforts to Enhance First Responders' All- Hazards Capabilities Continue to Evolve. GAO-05-652. Washington, D.C.:  July 11, 2005.
Results-Oriented Government: Improvements to DHS's Planning Process  Would Enhance Usefulness and Accountability. GAO-05-300.  Washington,  D.C.: March 31, 2005.
Homeland Security: Agency Plans, Implementation, and Challenges  Regarding the National Strategy for Homeland Security. GAO-05-33.   Washington, D.C.: January 14, 2005.
Homeland Security: Process for Reporting Lessons Learned from Seaport  Exercises Needs Further Attention. GAO-05-170. Washington, D.C.:  January 14, 2005.
Homeland Security: Federal Leadership and Intergovernmental  Cooperation Required to Achieve First Responder Interoperable  Communications.  GAO-04-740. Washington, D.C.: July 20, 2004.