Catastrophic Planning: States Participating in FEMA's Pilot Program Made Progress, but Better Guidance Could Enhance Future Pilot Programs
Published: 2011-04-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Hurricane Katrina in 2005 highlighted gaps in the nation's preparedness to respond effectively to catastrophic incidents. The Department of Homeland Security's (DHS) Federal Emergency Management Agency (FEMA) designed the Task Force for Emergency Readiness (TFER) pilot program to advance and integrate state and federal catastrophic planning efforts. TFER, first envisioned by the Department of Defense (DOD), ran from September 2008 to September 2010, and FEMA provided the five participating states--Hawaii, Massachusetts, South Carolina, Washington, and West Virginia--with $350,000 each to develop plans, build relationships with stakeholders, and document lessons learned (i.e., TFER's stated objectives). As requested, GAO evaluated the extent to which (1) FEMA followed sound management practices in designing, administering, and evaluating TFER and (2) the five participating states satisfied TFER's stated objectives. GAO analyzed FEMA guidance, such as the TFER Pilot Information Package, conducted site visits to all five participant states, and met with relevant FEMA and DOD officials, to evaluate FEMA's management of TFER and the states' implementation of it.
section_title: What GAO Found
paragraphs: FEMA developed program objectives and procedures for administering the TFER pilot, but did not develop other elements of sound management practices in designing, administering, and evaluating pilot programs that GAO identified from its prior work and social science literature. FEMA developed objectives for the pilot, but did not document standards for determining the pilot's success. FEMA also provided resources such as funding, training, and support, but FEMA did not always follow the procedures it established for TFER. For example, FEMA did not consistently conduct biweekly conference calls with the states, and four states reported that it would have been helpful if FEMA provided more guidance. FEMA did not develop a data analysis plan, which could have better ensured FEMA collected data on the extent to which the pilot states coordinated with key stakeholders and provided reasonable assurance that FEMA conducted a systematic assessment of TFER using comparable data across the five pilot states. In Spring 2010, FEMA announced TFER would not continue past the pilot stage before evaluating TFER on its merits in strengthening and advancing state catastrophic planning, but FEMA allowed the states to draw down the remaining TFER funds and continue to complete initiatives started under TFER through August 2011. GAO previously reported in April 2009 that FEMA faced challenges in assessing pilot program data, and FEMA officials reported the agency does not have pilot program policy guidance. In the absence of this, FEMA lacks a systematic approach to developing, administering, and evaluating pilot programs. FEMA could better ensure other pilot programs meet their intended goals by developing policies and guidance that include sound management practices. All five states have taken steps to follow FEMA's guidance to address TFER's objectives, but no state has fully addressed them all. First, two of the five states have completed draft catastrophic plans, and all five states reported following FEMA's planning process. Second, all five states built relationships with stakeholders such as state agencies and FEMA, but state officials said coordination with DOD--a key federal stakeholder who may be called upon to assist in disaster response--was limited. State officials reported not coordinating with DOD because they did not have draft plans for DOD officials to review. Third, all states have documented lessons learned to date, but four states have not exercised TFER plans to determine their effectiveness in the event of an emergency.
section_title: What GAO Recommends
paragraphs: GAO recommends that FEMA develop policies and guidance that follow sound management practices for future pilot programs, and share TFER results with stakeholders. FEMA agreed with GAO's recommendations.

section_title: Letter
paragraphs: Hurricane Katrina in 2005 highlighted gaps in the nation’s preparedness to  respond effectively to catastrophic incidents. In the aftermath of  Hurricane Katrina, the Federal Emergency Management Agency (FEMA), a  component of the Department of Homeland Security (DHS), was charged  with leading the nation in preparing for disasters in coordination with  other federal entities like the Departments of Defense (DOD) and Justice;  the 50 states, the District of Columbia, and five territories; tribal, city, and  county governments; nonprofit organizations; and private entities. The  Department of Homeland Security Appropriations Act, 2010 appropriated  approximately $7.1 billion to FEMA, and FEMA has allocated more than  $1.8 billion to preparedness grant programs.
The Task Force for Emergency Readiness (TFER) pilot program was  launched by FEMA in September 2008 to strengthen state preparedness for  catastrophic disasters by facilitating greater capacity in and more  comprehensive integration of planning efforts across all levels of  government through $1.75 million in grant funds. In particular, TFER  emphasized integration of planning efforts across sectors, jurisdictions,  and functional disciplines, as well as integration among state, regional, and  federal agencies, primarily FEMA and DOD. The TFER grant funds  allowed states to hire three full-time planners with civilian and military  planning expertise to develop catastrophic plans. FEMA selected five  states to participate in this pilot program: Hawaii, Massachusetts, South  Carolina, Washington, and West Virginia.
Planning is a key component of national preparedness. According to  FEMA planning guidance, planning provides a methodology to examine  the entire life cycle of a potential crisis, determine required capabilities,  and help stakeholders learn and practice their roles for building and  sustaining national preparedness capabilities against terrorist attacks and  other hazards. This is particularly important in a catastrophic incident,  defined as any natural disaster, act of terrorism, or man-made disaster that  results in extraordinary levels of casualties or damage or disruption  severely affecting the population, infrastructure, environment, economy,  national morale, or government functions in an area. Catastrophic events  differ from a normal disaster in the severity of the damage, number of  persons affected, and the scale of preparation and response required. They  quickly overwhelm or incapacitate local and/or state response capabilities,  thus requiring coordinated assistance from outside the affected area.
You asked us to review the TFER pilot program. Our objectives for this  report were to identify the extent to which (1) FEMA followed sound  management practices in developing, administering, and evaluating the  TFER pilot program and (2) the five participating states satisfied the TFER  pilot program’s stated objectives.
To conduct this work, we completed site visits to all five participant states  from June through September 2010 and interviewed relevant FEMA, DOD,  and DHS headquarters and regional officials. During our site visits, we met  with the planners hired through the TFER pilot program and other state,  local, and DOD and FEMA regional officials to discuss FEMA’s  management of the TFER pilot program and the states’ implementation of  it.
To identify the extent to which FEMA followed sound management  practices in developing, administering, and evaluating TFER, we analyzed  program guidance and other key documents including the TFER Pilot  Information Package and the TFER Guidance and Application Kit. We also  analyzed our prior work on pilot programs and program management as  well as social science literature to determine elements of sound  management practices. We grouped these elements into three categories:  design, administration, and evaluation. We analyzed the program guidance,  other documents, and information obtained from our interviews and site  visits and compared the results of our work with the identified elements of  sound management practices. Furthermore, we analyzed our prior work  and DHS Office of Inspector General reports on FEMA grants  administration and catastrophic preparedness to identify any past  challenges.
To identify the extent to which the five states participating in TFER  addressed the program’s stated objectives, we analyzed information that  FEMA provided in the TFER Pilot Information Package and further  defined in its TFER Evaluation Criteria document. The three program  objectives included developing plans, building relationships, and  documenting lessons learned, and FEMA provided criteria to measure  progress towards addressing each of these objectives. We used FEMA’s  criteria to then assess states’ progress toward addressing the three  program objectives. Additionally, FEMA identified key components of  each objective. To assess the states’ progress towards addressing TFER’s  program objectives, we assigned a numerical ranking to each objective’s  components and then averaged the components’ numerical rankings for  each of the objectives. We then assigned designations to these averages.  We determined that states could not receive a designation of “substantial  progress” for their objectives without meeting the criteria related to these  key components. To assist in assessing states’ progress, we reviewed and  analyzed the documents obtained during our site visits that included draft  plans, progress reports, and status briefings, and interviewed cognizant  officials. As part of this analysis, we reviewed the guidance in FEMA’s  Comprehensive Preparedness Guide (CPG) 101, which provides  instructions regarding plan structure, and we used this information to  determine the extent to which the states’ plans adhered to FEMA’s  guidance.
We conducted this performance audit from May 2010 to April 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. Additional details on our scope and  methodology are contained in appendix I.
subsections: section_title: Background
subsections: section_title: The National Response Framework Established a Tiered Response for Emergencies
paragraphs: DHS issued the National Response Framework for federal, state, and local  agencies to use in planning for emergencies.  It establishes standardized  doctrine, terminology, and processes for responding to disasters and other  catastrophic events in the United States. The framework is based on a  tiered, graduated response; that is, incidents are managed at the lowest  jurisdictional levels and supported by additional higher-tiered response  capabilities as needed (see fig. 1).
Local governments respond to emergencies using their own resources or  assistance from neighboring localities. For larger-scale incidents that  overwhelm the capabilities of local governments, assistance from the state  may be sought. Depending on the circumstances, states have capabilities,  such as the National Guard, that can help communities respond and  recover. If additional resources beyond what an individual state can  provide are required, the state may request assistance from other states  through Emergency Management Assistance Compacts or from the federal  government. Catastrophic incidents, by definition, result in extraordinary  levels of damage or disruption to government functions, and governors  have the option to seek federal assistance. The federal government has a  wide array of capabilities and resources to assist state and local agencies  respond to incidents. In accordance with the National Response  Framework and applicable laws including the Robert T. Stafford Disaster  Relief and Emergency Assistance Act (Stafford Act), various federal  departments and agencies may play primary, coordinating, or supporting  roles, based on their authorities and resources and the nature of the  incident. In certain instances, national military capabilities may be  requested to respond to an incident. Defense resources are committed  following approval by the Secretary of Defense or at the direction of the  President.
According to FEMA planning guidance, plans must be integrated  vertically among levels of government (e.g., local, state, and federal) to  ensure a common operational focus and horizontally (e.g., health  departments and law enforcement) to ensure that individual department  and agency operations plans complement the jurisdiction’s plans. This  integration enables stakeholders to synchronize the sequence and scope of  a planned response in terms of roles and responsibilities, place, and time  to a catastrophic incident. Catastrophic incidents include those that are  defined in the National Planning Scenarios, which represent examples of  the gravest dangers facing the United States, including terrorist attacks  and natural disasters, and have been accorded the highest priority for  federal planning efforts.
section_title: FEMA and DOD Each Play a Role in Emergency Response
paragraphs: FEMA’s primary mission is to reduce the loss of life and property and  protect the nation from all hazards, including natural disasters, acts of  terrorism, and other man-made disasters, by leading and supporting the  nation in a risk-based, comprehensive emergency management system of  preparedness, protection, response, recovery, and mitigation. As  described in the National Response Framework, planning is one of six  activities essential to preparing for an incident. Other activities include  training; equipping; exercising; and evaluating and improving to build  tribal, local, state, regional, and national capabilities necessary to respond  to any type of disaster.
As shown in figure 2, FEMA has 10 regional offices nationwide. Each  region serves several states, and FEMA regional personnel work generally  with the states to help plan for disasters, develop mitigation programs, and  meet other needs when major disasters occur. Regional office locations  are starred in the figure below, and in some cases are located within a  pilot state.
Washington, D.C.
U.S. Northern Command (NORTHCOM) and U.S. Pacific Command  (PACOM) are the combatant commands charged with carrying out DOD’s  domestic civil support mission, which involves responding to the needs of  56 separate and often unique state and territorial governments.  Additionally, NORTHCOM and PACOM coordinate with numerous federal  agencies that also have roles in planning for and responding to a variety of  incidents in the homeland. As part of the lessons learned from Hurricane  Katrina, both NORTHCOM and PACOM have established Defense  Coordinating Officers with staff to serve as DOD representatives to civilian  authorities in the 10 FEMA regions. Defense Coordinating Officers are  full-time senior-level military officers who provide liaison support and  requirements validation, and they serve as single points of contact for  state, tribal, local, and other federal authorities that need DOD support. In  that way they are able to develop relationships with civilian authorities  and gain an understanding of civilian capabilities so that DOD will know  what, if anything, it may be called upon to provide in the event of a  disaster or other incident. Additional defense resources include  Emergency Preparedness Liaison Officers who are senior Reserve officers  that are expected to assist the Defense Coordinating Officers in  coordinating the provision of military personnel, equipment, and supplies  to support the emergency relief and cleanup efforts of civilian  authorities. When NORTHCOM and PACOM are called upon to support  civilian authorities, in most cases support will be localized, limited, and  specific. When the scope of the disaster is reduced to the point where the  primary federal agency can again assume full control and management  without military assistance, NORTHCOM and PACOM will withdraw.
section_title: DOD Envisioned TFER, but FEMA Implemented the Pilot Program
paragraphs: The TFER pilot program was first envisioned by senior DOD leadership to  address the National Planning Scenarios through supporting and  strengthening the catastrophic disaster preparedness of individual states.  However, since DOD is always a supporting agency when responding to  catastrophic events in the homeland, DOD leadership did not think it was  appropriate for DOD to administer the pilot program, and FEMA  administered and funded the pilot program. Although FEMA administered  TFER, pilot program guidance stated that the TFER program was to place  a special emphasis on coordination and integration between the pilot  states, FEMA, and DOD. Further, a key measure of achievement in the  pilot program is the degree to which DOD stakeholders are integrated as  full partners in the development of states’ plans.
FEMA designed TFER to help assess, strengthen, and advance state  catastrophic preparedness planning across state and local agencies and  the private sector, as well as achieve a fuller integration of state planning  efforts with federal agency partners through a deliberate blending of civil- military planning expertise. The TFER pilot program centered on the  creation and employment of a dedicated planning team. Specifically, the  planning team was to synchronize catastrophic planning efforts at the  respective state, regional, and federal levels; prepare planning documents;  identify gaps in capabilities; and assist in the overall organization,  administration, and improvement of the states’ catastrophic preparedness  planning capacity. The three objectives of TFER are to (1) develop plans,  (2) build relationships, and (3) document lessons learned.
FEMA designed TFER to run 18 months, beginning on September 1, 2008,  and ending on March 31, 2010. The pilot program was extended another 6  months to September 30, 2010—for a total time of 2 years—to allow states  more time to accomplish their planning goals. As shown in figure 3 , FEMA  announced in March 2010 that TFER would not continue past the pilot  stage, but allowed the pilot states to draw down the remaining TFER funds  and continue to complete initiatives started under TFER through August  2011.
FEMA limited the pilot program to five states, each of which received  $200,000 in the first year of the pilot program, and an additional $150,000  for the remainder of the program, for overall program funds totaling $1.75  million. It selected applicants to achieve a representative cross-section of  geographic differences, FEMA regions, threat of catastrophic events, state  emergency management structures, and participation in a separate FEMA  catastrophic planning grant. As discussed in appendix II, FEMA provides  funds to states and localities for catastrophic planning through other grant  programs. For example, FEMA’s State Homeland Security Program  provides funding that may be used for planning efforts that enable states  to prioritize needs, build capabilities, update preparedness strategies,  allocate resources, and deliver preparedness programs across disciplines  and levels of government.
FEMA administers the pilot program within its National Preparedness  Directorate. This directorate is responsible for overseeing the  coordination and development of strategies necessary to prepare for all  hazards. As part of this mission, the National Preparedness Directorate is  to provide policy and planning guidance, training and exercises, and  technical assistance that builds prevention, protection, response, and  recovery capabilities.
In the aftermath of Hurricane Katrina, FEMA began to address the  challenges revealed by its response. FEMA’s 2006 Nationwide Plan Review  highlighted the need for fundamental planning modernization and  observed, among other things, that: catastrophic planning efforts were  unsystematic and uneven; planning expertise was insufficient for  catastrophic incidents; and collaboration requirements were not well- defined, fostering a tendency to plan internally. In addition, according to  FEMA, the 2006 Nationwide Plan Review revealed that 95 percent of state  and urban area participants cited the need for federally funded planning  support and technical assistance. FEMA also reported that in 2007, states  and urban areas ranked planning among their highest grant funding  priorities. The Department of Homeland Security’s Office of Inspector  General recently reported that while progress has been made, the National  Preparedness Directorate needs more effective coordination with state,  local, and tribal governments. Our prior work on FEMA, and specifically  the National Preparedness Directorate, has highlighted the need for  strategic planning that identifies outcomes and performance measures for  the Directorate’s programs. In October 2010, we also reported that since  April 2009, FEMA has made limited progress in assessing preparedness  capabilities and has not yet developed national preparedness capability  requirements based on established metrics to provide a framework for  these assessments. We reported that until such a framework is in place,  FEMA will not have a basis to operationalize and implement its conceptual  approach for assessing local, state, and federal preparedness capabilities  against capability requirements to identify gaps for prioritizing  investments in national preparedness. FEMA responded that it has made  much progress since 2009 in meeting its legislative requirements and  highlighted some of its specific achievements, such as the establishment of  a working group to help consolidate and streamline reporting  requirements for state, tribal, and local stakeholders.
section_title: FEMA Implemented Some, but Not All Elements of Sound Management Practices in Its Administration of the TFER Pilot Program, and Does Not Have Policy Guidance on Developing Pilot Programs
paragraphs: FEMA developed program objectives and a data collection plan—elements  of sound management practices—but did not implement other elements of  sound management practices, such as documenting standards for  determining the program’s success and did not always follow its stated  processes and procedures for TFER. Further, FEMA does not have  agencywide policy guidance for FEMA program managers to follow when  developing pilot programs. In November 2008, we reported that pilot  programs can more effectively inform future program rollout when sound  management practices are followed. Consistent with best practices in  program management, our guide for designing evaluations, and our  prior work, we identified sound management practices to design a pilot, to  guide consistent implementation of a pilot, and to conduct analysis of the  results. Our assessment of FEMA’s implementation of sound management  practices for TFER is shown in table 1.
subsections: section_title: FEMA Developed Objectives that Reflect the Goals of the Pilot Program, but Did Not Document Standards for Determining Pilot Program Success
paragraphs: FEMA developed objectives for the pilot states to follow, but did not  document standards for determining whether TFER was successful.  According to sound management practices, having objectives that reflect  the goals of the pilot program can help determine whether the objectives  of the pilot have been met. FEMA outlined three objectives for the TFER  pilot program—develop plans, build relationships, and document lessons  learned—and these objectives reflect TFER’s goals. For example, the  “develop plans” and “document lessons learned” objectives reflect the  goals of TFER because achieving these objectives may help assess,  strengthen, and advance state catastrophic preparedness planning—a  stated TFER goal. Further, achieving the “build relationships” objective  may enhance the stated TFER goal of integrating state planning efforts  with those of federal agency partners. However, FEMA did not define how  states’ implementation of the objectives would impact the decision to  extend or end TFER. According to sound program management practices,  defining standards for determining pilot-program performance is  necessary to determine what success level is appropriate for judging the  pilot program’s effect. The TFER program manager said that discussions  about the expectations for TFER were held at a high level with senior  leadership from FEMA, DOD, and the participating states, such as the  Adjutants General, before TFER planners were hired. However, FEMA did  not include the standards resulting from these conversations in the TFER  program guidance. The program manager said that the agency had a  limited time frame in which to design the pilot and that if he were to  design the TFER pilot now, he would build language into the grant  guidance that included standards for measuring performance. By clearly  defining standards for determining program performance, FEMA would be  better positioned to determine whether TFER enhanced states’  catastrophic planning, as well as achieved a fuller integration of state  planning efforts with federal agency partners through a deliberate  blending of civil-military expertise.
section_title: FEMA Outlined Steps to Administer TFER, but Did Not Always Adhere to Its Processes and Procedures
paragraphs: FEMA outlined steps for administering TFER, which aligns with sound  management practices, but FEMA did not always adhere to the processes  and procedures it developed for the pilot. Sound management practices  include developing processes and procedures for approving, reporting,  and monitoring and providing program support. Further, according to  Standards for Internal Control in the Federal Government, such control  activities are an integral part of an entity’s planning, implementing,  reviewing, and accountability for stewardship of government resources  and achieving effective results. FEMA outlined steps in two key  documents, the TFER Guidance and Application Kit and the TFER Pilot  Information Package in which FEMA described the support it would  provide the participating states and the procedures for monitoring the  program.
FEMA developed processes and procedures for providing funding,  training, holding a meeting with the pilot states, and approving state  project management plans that met certain requirements, and establishing  a mechanism for sharing lessons learned among the states, but FEMA did  not fully follow the steps it outlined. For example, FEMA provided $1.75  million in funds to the state administrative agencies identified in the TFER  Pilot Information Package and established a virtual communication tool  for the TFER pilot states to communicate and share lessons learned.  However, FEMA provided training for the initial cohort of TFER planners  in April 2009, which FEMA officials told us consisted of two-thirds of the  planners originally hired by the states. All five states found FEMA’s  training useful. However, officials in two states told us additional training  would have been useful. For example, one TFER planner who was not  hired at the time of the training stated that additional training could have  provided planners more opportunities to network and gain knowledge  about state and federal resources. Further, FEMA conducted a meeting  with the pilot states to accomplish tasks such as establishing support  requirements and planning deliverables, but FEMA approved three state  project management plans that did not contain all the recommended  information, such as specific tasks, milestones, and deliverables. The  TFER program manager stated that part of the intent of TFER was to  design it in real time, working with the jurisdictions, and that the  omissions of the recommended information did not negatively impact the  states’ implementation of TFER. However, the states whose project  management plans did not include specific deliverables have not  completed draft plans.
Further, FEMA also established processes and procedures for reporting  and monitoring, but did not fully follow them. Specifically, FEMA stated in  the TFER Pilot Information Package that the project manager would  conduct biweekly conference calls with the states, but state officials in all  five pilot states noted that FEMA headquarters involvement was  inconsistent. According to the program manager, the purpose of these  meetings was to ensure the pilot was functioning smoothly, so FEMA  conducted biweekly meetings as TFER became operational but the  meetings were discontinued once the pilots were underway. However,  officials in four states reported that it would have been helpful if FEMA  had provided more guidance on plan development. Further, TFER  planners in one state reported that the lack of guidance made it more  difficult to write plans. Another planner stated that FEMA did not provide  the expected level of meetings and technical assistance. The program  manager stated he was available to provide the states with assistance as  needed throughout the pilot program. In addition, FEMA established  reporting requirements and a plan to conduct a 6-month evaluation. For  example, the states submitted reports to FEMA—to include the status of  milestones and deliverables—but the program manager told us he did not  review all of these reports because not all of these reports were submitted  to his office. As a result, the program manager could not monitor progress  in meeting milestones and completing deliverables. FEMA also conducted  a 6-month evaluation of the pilot program, but four pilot states reported  that FEMA did not provide feedback on the evaluation. According to  sound management practices, continuous monitoring can be a useful tool  for facilitating effective management by providing regular feedback about  how well a program is performing its functions. This type of feedback  allows managers to take corrective action when problems arise and can  also provide stakeholders with regular assessments of program  performance. By adhering to the steps outlined in TFER documents,  FEMA could have more effectively monitored the pilot program to better  ensure that the states receive the level of guidance and support they  identified as beneficial in order to enhance their TFER catastrophic  planning efforts.
section_title: FEMA Took Steps to Collect and Analyze Data from States to Evaluate TFER, but Did Not Always Include Elements of Sound Management Practices
paragraphs: FEMA developed a data collection plan but did not reliably collect data for  the pilot evaluation, and FEMA did not develop a data analysis plan to  describe how it would track TFER’s performance and evaluate its  effectiveness based on standards to determine program performance. For  example, FEMA developed elements of a data collection plan, following  sound management practices such as detailing the type of data necessary  to monitor and evaluate the pilot and identifying the timing and frequency  of data collection. Further, FEMA’s data collection plan followed reporting  requirements specified in its program documents for TFER participants,  and also included a data collection tool for evaluation of the pilot,  presented in appendix III. However, FEMA did not always identify the  source that was to collect the data. Sound management practices say that  the source of data should be specified in the data collection plan, and  according to program evaluation guidance, the reliability of evaluations  can vary depending on who responds. During interviews with officials  from participating states and FEMA regional offices, we found  inconsistencies in who completed the evaluation questions. For example,  two pilot states developed their own responses and three FEMA regions  developed the responses for their respective pilot states, even though one  of the questions explicitly asked for the opinion of state stakeholders.  Further, two of the three states were not cognizant of the fact that their  FEMA regional offices had completed the evaluation questions; therefore,  these states’ views may not have been accurately captured in the  evaluation responses. Moreover, officials in one FEMA regional office  questioned whether they were the appropriate FEMA personnel to  administer the evaluation questions, since FEMA headquarters had  collected all prior information from the states. Collecting data from  consistent sources across the five pilot states could have better positioned  FEMA to ensure the data were reliable and compare responses across  states in order to draw conclusions about states’ experiences in the pilot  program.
Further, FEMA did not collect information to determine whether all of the  pilot program objectives were fully met. Specifically, FEMA changed some  of the data collection questions it provided the states in the TFER Pilot  Information Package when it released updated questions in the TFER  Evaluation Criteria document. For example, FEMA used the same three  objectives in both its original and updated documents, and its objectives  on developing plans and documenting lessons learned objectives were  consistent across documents, but FEMA did not collect information to  assess all of the relationships it identified as important for its objective on  building relationships in the TFER Pilot Information Package. For  instance, FEMA did not evaluate coordination with FEMA, Defense  Coordinating Officers, or Emergency Preparedness Liaison Officers—all  relationships principal to implementing TFER, according to the TFER  Pilot Information Package document. FEMA included a question that  asked the pilot states which federal agencies they coordinated with, but  did not include any question that would allow it to evaluate the extent to  which coordination occurred with these stakeholders. As a result, the  evaluation responses we reviewed listed the stakeholders they  coordinated with but did not discuss the extent of coordination between  state officials and stakeholders. FEMA officials said that because the  Defense Coordinating Officer and Emergency Preparedness Liaison  Officer positions were not the only relationships explored between states  and federal agencies, the evaluation criteria was changed to make it more  general to determine with which agencies the TFER states engaged.  However, by changing the evaluation question to a general and descriptive  question on federal relationships, it will be difficult for FEMA to determine  if the states achieved a fuller integration of planning efforts with federal  agency partners—one of the goals of TFER.
FEMA did not develop a data analysis plan to describe how it would track  TFER’s performance and evaluate its effectiveness based on criteria to  determine program performance. According to sound management  practices, a data analysis plan sets out who will do the analysis and when  and how data will be analyzed to measure the pilot program’s  performance. FEMA did not finalize a data-analysis methodology  describing how FEMA would use information collected from the states to  evaluate TFER until after it collected responses to FEMA’s evaluation  criteria questions. The TFER program manager stated that FEMA plans to  look for trends in the states’ responses to the evaluation questions to  discern common threads and outliers. According to the program manager,  FEMA did not develop a plan that outlined the overall approach it would  use to analyze the TFER data during design of the pilot program because  the pilot was not intended to produce quantitative results. He further  stated that FEMA wanted to see what worked and what did not work as  the pilot evolved. However, a data analysis plan could have helped guide  the analysis of the qualitative data FEMA collected from the pilot states  and FEMA regional offices. Developing data collection and data analysis  plans following sound management practices to establish a methodology  for collecting and evaluating the pilot’s results could have provided FEMA  with reasonable assurance that it possessed the data and performance  information needed to draw reasonable conclusions on the impact of  TFER.
The TFER program manager told us he is analyzing the data collected for  TFER to draw conclusions regarding the pilot program, but FEMA officials  stated that TFER would not be continued past the pilot stage. The  information on the TFER pilot program’s evaluation stated that the results  of the evaluation would determine the prospect for the program’s  continuation and level of future funding. However, prior to conducting its  evaluation of the pilot, FEMA announced in March 2010 that TFER would  not be continued as a stand-alone grant program. According to FEMA  officials, the decision to discontinue TFER as a stand-alone grant program  was made because DOD may begin a similar initiative and states could  allocate other FEMA grant funding for catastrophic planning purposes.  Specifically, the TFER program manager noted that the National Guard  Homeland Response Force, which is a DOD concept that would place  National Guard personnel in each FEMA region, could continue the  planning efforts associated with TFER. DOD officials stated that these  personnel would be located regionally and not in individual states;  therefore, the National Guard Homeland Response Force initiative could  supplement but not replace a TFER-like initiative. Also, according to DOD,  the National Guard Homeland Response Force would have as its mission  response to chemical, biological, radiological, nuclear, and high-explosive  incidents and would not have a broad purview like TFER. A FEMA  assistant administrator also said that FEMA made the decision to not  continue TFER due to changing budget conditions and to focus on its  legislative and presidential directive mandates. Other FEMA grant  programs available for catastrophic planning are discussed in appendix II.
section_title: FEMA Has Not Committed to Sharing the TFER Evaluation Results with Stakeholders
paragraphs: According to the TFER program manager, FEMA leadership will make a  decision on whether and how the results of the evaluation are  disseminated after the program office finishes the evaluation of TFER.  According to sound management practices, results and recommendations  that emerge from evaluations must be disseminated in ways that meet the  needs of stakeholders. The TFER program manager stated it is not his  decision whether the results of the pilot’s evaluation will be shared.  According to the TFER program manager, FEMA leadership will make a  decision on whether and how the results of the evaluation are  disseminated after the program office finishes the evaluation of TFER.  FEMA administered the TFER pilot program, but other key stakeholders in  federal disaster response, such as the pilot states and DOD, could benefit  from the results of FEMA’s evaluation of the TFER pilot program.  Disseminating the results, recommendations, and lessons learned from the  TFER pilot could assist future state catastrophic planning efforts, whether  they be led by FEMA, DOD, another federal agency, or a state.
section_title: FEMA Does Not Have Clear Policy Guidance on Designing, Administering, and Evaluating Pilot Programs
paragraphs: FEMA’s Office of Policy and Program Analysis reported that FEMA does  not have a directive on how to design, administer, and evaluate a pilot  program. In the absence of this, FEMA does not have a systematic  approach for designing, administering, and evaluating such programs. The  program manager stated there are no formal guidelines for developing  FEMA grants. We previously reported that FEMA faced methodological  challenges in assessing capabilities and has not generated meaningful  preparedness information from data collected from pilot programs.  Standards for Internal Controls in the Federal Government state that  management is responsible for ensuring that detailed policies, procedures,  and practices are developed and built into and are an integral part of  operations. These elements, which are an integral part of an agency’s  ability to ensure accountability and achieve effective results, need to be  clearly documented to help ensure that management directives are carried  out as intended. By developing and implementing policy guidance that  includes sound management practices, FEMA could be better positioned  to ensure its pilot programs meet their intended goals.
section_title: All Pilot States Have Made Progress in Addressing the Pilot Program Objectives, but No State Has Fully Addressed All Three Objectives
paragraphs: All five states have taken steps to follow FEMA’s TFER Information  Package and TFER Evaluation Criteria document to address the TFER  pilot program’s three objectives—develop plans, build relationships, and  document lessons learned—and these efforts are ongoing. First, of the  four states committed to developing plans, two states have completed  draft plans, and TFER planners in all states reported using FEMA’s CPG  101 planning process to develop initiatives that fill gaps in state  catastrophic planning. Second, all five states built relationships with  stakeholders, including FEMA and state agencies, but coordination with  DOD—specifically, with the Defense Coordinating Officers and  Emergency Preparedness Liaison Officers—was limited. Third, all states  documented lessons learned from the pilot program, which included the  benefit of dedicated funding for catastrophic planning, and establishing  longer time frames for states to complete the planning process. However,  these efforts are ongoing as four states continue to spend TFER funds.
subsections: section_title: Two of Five States Have Made Substantial Progress in Developing Plans, and TFER Planners in All States Reported Using FEMA’s Planning Process to Develop Initiatives that Fill Gaps in State Catastrophic Planning
paragraphs: Four of the states are developing plans and all five pilot states reported  they are using the planning process outlined in FEMA’s CPG 101.  However, two of the four states developing plans have not completed  them. The fifth state is focusing on other planning initiatives. FEMA  articulated the criteria for this objective in the TFER Pilot Information  Package and the TFER Evaluation Criteria document, which focused on  organizing plans according to CPG 101 and plan development. Based on  audit work conducted in the TFER pilot states, we assessed the states’  progress based on these criteria, and concluded that two of the states have  made substantial progress in fulfilling the FEMA objective on developing  plans, and the other three have made some progress, as shown in figure 4.  Figure 4 further explains this assessment, as well as appendix IV, which  provides specific information regarding each pilot state’s TFER initiatives  and progress made towards meeting FEMA’s objectives for TFER.
Developing plans. The status of the states’ TFER plans varies; Washington  and South Carolina have completed some but not all of their plans, while  Hawaii and West Virginia have not completed any draft plans. Hawaii and  West Virginia officials identified reasons for not having completed draft  plans, including delays in hiring qualified planners. One of FEMA’s  requirements for TFER was that planners have experience in conducting  civil-military support planning and operations with the National Guard  and/or military experience. In Massachusetts, South Carolina, and  Washington, all planners initially hired for TFER had military  backgrounds. These states hired planners as early as February 2009—5  months after the pilot program began. One of the planners initially hired  had prior military and civilian experience, and the other had a logistics  background. Hawaii hired its first TFER planner in April 2009 but did not  hire a second planner until August 2009—11 months after the pilot began.  Hawaii officials told us that they found it difficult to hire qualified  individuals with a military background because better paying options exist  in the state for qualified individuals. One of Hawaii’s two current planners  does not have military experience, but state officials told us that the  individual is qualified because of his information technology background,  which was needed to support logistics planning. Additionally, he worked  for the state planning agency. West Virginia officials reported that they  hired their planners in the July/August 2009 time frame—10 to 11 months  after the pilot began. Two applicants originally applied for the position,  one of which West Virginia hired. This individual did not have a military  background, but officials stated he was qualified based on his decades of  experience as a first responder. West Virginia subsequently hired a  National Guardsman. The West Virginia TFER supervisor stated that  inadequate planning capacity at the local level also delayed TFER efforts  because West Virginia TFER focused on building capacity at the local  level. FEMA granted all five states a 6-month extension based on delays  the states experienced. As previously discussed, planning continues in  four states, as they are still spending TFER funds, as of March 2011 (see  fig. 5).
Massachusetts TFER planners facilitated planning between different levels  of government, but other state planners are developing the plans.  Massachusetts initially focused on contributing to Boston’s improvised  explosive device planning efforts, but differences in time lines, combined  with TFER’s brief performance period, caused the team to change its focus  to facilitating collaborative emergency planning among federal, state,  local, and private stakeholders. For example, the Massachusetts TFER  planners facilitated collaborative planning among the Massachusetts  Department of Public Health, the U.S. Department of Health and Human  Services, the Massachusetts Emergency Management Agency, and FEMA  to create a statewide medical support plan. TFER planners developed  interagency relationships and helped the planning team select goals and  objectives but did not draft the plan, which state officials said other state  planners are developing. State officials decided the TFER planners would  not actually write the plans because of the pilot program’s short time  frame.
Filling gaps in state planning. State officials reported that TFER  initiatives fill gaps in existing state plans. For example, South Carolina,  Washington, and West Virginia chose to work on a catastrophic incident  plan as part of their TFER efforts, and this plan is an annex to the states’  basic emergency operations plan. Further, officials in four of five states  reported that TFER initiatives focused on gaps not addressed by other  planning initiatives. For example, the FEMA Region X TFER contact  stated that Washington successfully integrated TFER with the Regional  Catastrophic Grant Planning Program. Two of three West Virginia TFER  scenarios pertain in part to scenarios addressed by other planning efforts,  but officials stated that TFER planners work collaboratively with the  planners involved in these other efforts, as described in appendix IV.  Additionally, Hawaii officials anticipate the Regional Catastrophic  Planning Grant Program will address hurricanes in 2011—before the state  exhausts its TFER funds. State officials expect these efforts to be  complementary.
FEMA’s planning process. All five states reported that they are following  FEMA’s planning process, and states that developed draft plans utilized  the structure recommended in FEMA’s CPG 101 (see fig. 6). All of the  states reported they are following FEMA’s six-step planning process, but  they are in different stages of that process, as of March 2011.
Washington has exercised three out of seven of its plans—part of step six.  However, most of Washington’s plans remain in draft form, which is  associated with step five. Since Hawaii, South Carolina, and West Virginia  officials told us their states have not exercised any plans, they are in step  five—the step where planners write the plan and the plan is approved.  Massachusetts state officials decided that their pilot program would  coordinate planning efforts, but that other state planners would develop  plans; officials stated they did not move beyond step four in the planning  process.
Officials in the four states that committed to developing plans reported  that they are structuring plans as recommended by CPG 101, but some  sections are not included in the states’ draft plans. CPG 101 provides a  recommended structure for annexes to a state’s basic emergency  operations plan.  These annexes comprise half of the 32 TFER initiatives  spanning all five pilot states. Ten of 13 annexes we reviewed were missing  at least one recommended section. For example, none of the three  catastrophic incident plans we reviewed contain a section on direction,  control, and coordination. South Carolina and Washington officials noted  that their plans did not include all of the recommended sections because  other state plans like the basic emergency operations plan contained these  sections. Washington officials further stated that TFER-supported  planning is part of the state emergency operations plan and is not designed  to stand alone. Further, Hawaii officials told us that they are following the  FEMA guidance, but its plans are incomplete and work is ongoing. West  Virginia officials stated that it is following the structure recommended in  CPG 101, but since some of the sections rely on information gathered at  the county level, it will take about 5 years before the state has enough  information to include each recommended section in its catastrophic  incident plan. We were unable to further assess the content of the pilot  states’ plans because all but three of the plans TFER states are developing  remain in draft form.
section_title: All Five States Made Progress in Building Relationships with Stakeholders such as FEMA and State Agencies, but Coordination with Federal Military Stakeholders Was Limited
paragraphs: relevant stakeholders. As discussed in figure 7, all five states have made  some progress in building relationships with relevant stakeholders. Our  assessment—based on audit work conducted in the TFER pilot states—is  summarized in figure 7 and further explained below as well as in  appendix IV, which provides specific information regarding each pilot  state’s TFER initiatives and progress made towards th is objective.
section_title: All Five Pilot States Have Made Some Progress in Documenting Lessons Learned, but Four States Have Not Exercised Plans to Evaluate TFER’s Effectiveness
paragraphs: All states have documented lessons learned to date, but four states have  not exercised TFER plans to determine their effectiveness in the event of  an emergency. As shown in figure 8 and further explained in appendix IV,  the TFER pilot states have provided a number of lessons learned and  recommendations on how to improve a similar program in the future.  One state, Washington, has made substantial progress documenting  lessons learned because it has exercised three of its plans to evaluate their  effectiveness, one of the components FEMA cited in the objective. FEMA  articulated the criteria for this objective in the TFER Pilot Information  Package and the TFER Evaluation Criteria document, which focused on  documenting lessons learned, including exercising plans to evaluate  TFER’s effectiveness. All five pilot states documented lessons learned for  a report published by the National Guard Bureau in August 2010, and we  obtained additional lessons learned during our interviews with state  officials. Further, two of the states provided us with the lessons learned  they submitted to FEMA for its evaluation, which is not yet complete.
Some of the lessons learned were similar across the pilot states. For  example, all five states considered the pilot a success; Hawaii,  Massachusetts, South Carolina, and West Virginia officials reported that  TFER allowed their states to conduct catastrophic planning that otherwise  would not have occurred, and Washington reported that TFER advanced  its catastrophic and emergency logistics planning by at least 2 years.  Washington officials further stated that the $350,000 the state received in  TFER funding generated more return on investment than any other  $350,000 in the state’s emergency planning budget. Based on their view of  TFER’s success, officials in each of the five pilot states recommended  FEMA expand TFER to other states.
All states reported three principal reasons for considering TFER a success:  (1) autonomy to develop their own work plans, (2) provision of dedicated  funds to hire planners, and (3) the background of the TFER planners.  (1) State officials reported that they worked collaboratively with  FEMA to select catastrophic scenarios. For example, South  Carolina chose to work on plans related to terrorism attacks  because state officials believe they had adequately planned for  hurricanes and earthquakes—the two National Planning Scenarios  ranked higher than terrorism attacks in the state’s risk assessment.  Officials in all five states highlighted the benefit of this flexibility.  (2) Officials in all five states reported that the narrow focus of  TFER, dedicated solely to enhance catastrophic planning, was key  to the pilot program’s success. State officials told us that state  planners are often required to assist emergency planning and  response efforts during more routine disasters, such as seasonal  floods and forest fires, but since the terms of TFER did not allow  states to use TFER funds for these efforts, catastrophic planning  continued unabated.  (3) Officials in all five states reported that the planners’ military  background was beneficial. For example, state officials in  Massachusetts told us the planners’ military experience provided  them with a level of experience and training not often found in  other sectors and gave them credibility. However, officials in three  states told us that civilian planning or emergency management  experience also provided TFER planners the needed skill set.
States also reported challenges they faced, including (1) short time frames  that limited their ability to complete the planning process, and (2) not  enough guidance for the specialized plans they were developing.  (1) All five states reported that the short time frames limited their  ability to complete the planning process. FEMA structured TFER  as an 18-month program, and states noted that this was an  insufficient amount of time to complete the full cycle of planning.  Officials in one state thought that an adequate time frame would  have been 3 years, noting that this is the time frame FEMA  established for the Regional Catastrophic Planning Grant  Program. The TFER program manager stated that FEMA’s 6- month program extension—bringing the pilot program time frame  to 2 years—provided a sufficient amount of time to complete the  TFER initiatives.  (2) CPG 101 provides general format and content guidance  applicable to all plans, but officials in four of the pilot states said  that it would have been helpful for FEMA to provide guidance for  specialized plans such as catastrophic incident annexes, terrorism  plans, and logistics plans. For example, one TFER planner stated  that clearer guidance could have accelerated the planning  process. Further, officials in a fourth state said that they are using  an unpublished FEMA guide to support their planning efforts.  FEMA officials told us that states can request technical assistance  and examples of well-developed plans to aid in their own plan  development. According to the TFER program manager, FEMA is  in the process of developing additional Comprehensive  Preparedness Guides that will inform plan content for future state  planning efforts.
The process of documenting lessons learned is ongoing because Hawaii,  South Carolina, Washington, and West Virginia are still developing TFER  plans and spending TFER funds.
section_title: Conclusions
paragraphs: Hurricane Katrina highlighted gaps in the nation’s preparedness to  respond effectively to catastrophic incidents. By their nature, catastrophic  events involve extraordinary levels of casualties, damage, or disruption  that will likely immediately overwhelm state and local responders— circumstances that make sound planning for catastrophic events all the  more crucial. Planning is a key component of national preparedness.  Planning provides a methodology to determine required capabilities and  helps stakeholders learn and practice their roles for building and  sustaining national preparedness capabilities against terrorist attacks and  other hazards. As state and local governments continue to develop and  improve plans for catastrophic events and identify potential resource  shortfalls, the federal government will be in a better position to  understand the nature of the gaps it may be called upon to fill if state and  local resources are overwhelmed. The TFER pilot program was one such  effort to enhance catastrophic preparedness and provide federal  stakeholders with valuable information regarding local, state, or regional  response capability. However, by not consistently following sound  management practices to design, administer, and evaluate the TFER pilot  program and by not following its processes and procedures for  administering the pilot, it is unclear whether TFER achieved its intended  purpose. As we have reported, FEMA has piloted other efforts that have  not generated meaningful preparedness information from the data  collected. Future pilot programs at FEMA could benefit from policy  guidance that includes sound management practices to design, administer,  and evaluate pilot programs. In addition, future state catastrophic planning  efforts could benefit from the dissemination of the evaluation results of  the TFER pilot program to relevant stakeholders.
section_title: Recommendations for Executive Action
paragraphs: We recommend the Administrator, Federal Emergency Management  Agency, take the following two actions.
To help ensure future pilot programs achieve their intended results and  provide the performance information needed to make effective  management decisions for broader implementation, develop and  implement policies and guidance for pilot programs that follow sound  management practices. This guidance should include, at a minimum,  requirements for    a clearly articulated methodology with objectives reflective of  overall program goals and standards for determining program  performance;    procedures for monitoring program performance;    a data collection plan;    a data analysis plan; and    a process to disseminate the results and lessons learned that  emerge from the pilot.
To help ensure stakeholders receive valuable information regarding  catastrophic preparedness from lessons learned during the TFER pilot  program, disseminate the evaluation results and recommendations that  emerge from the TFER pilot program in ways that meet the needs of  current and future stakeholders.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to FEMA and DOD for review and  comment. FEMA provided oral comments on the draft report on April 5,  2011. FEMA fully concurred with both of our recommendations but did  not specify how it planned to address them. DOD did not provide  comments on the draft report because the report did not include  recommendations to DOD. Both FEMA and DOD provided technical  comments, which we incorporated throughout our report as appropriate.
As agreed, unless you publicly announce the contents of this report earlier,  we plan no further distribution until 30 days from the report date. At that  time, we are sending copies of this report to the Administrator, Federal  Emergency Management Agency, and Secretary of Defense. In addition,  the report will be available at no charge on GAO’s Web site at  http://www.gao.gov. If you or your staff have any questions concerning  this report, please contact Bill Jenkins at (202) 512-8777 or by e-mail at  jenkinswo@gao.gov, or Davi M. D’Agostino at (202) 512-5431 or by e-mail  at dagostinod@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.
section_title: Appendix I: Scope and Methodology
paragraphs: To identify the extent to which the Federal Emergency Management  Agency (FEMA) followed sound management practices in developing,  administering, and evaluating the Task Force for Emergency Readiness  (TFER) pilot program, we analyzed program guidance and other key  documents including the TFER Pilot Information Package and the  Guidance and Application Kit. We analyzed prior GAO work on pilot  programs and program management as well as social science literature  to determine elements of sound management practices. We considered  other criteria, consulting checklists on elements of program management,  but determined some of the characteristics identified in this literature  were not appropriate or suitable to the type of study performed by FEMA We grouped the elements of sound management practices that we  identified as relevant into three categories: design, administration, and  evaluation. We then analyzed the information obtained from our  interviews and site visits and compared the results of our analysis with th identified elements of sound management practices. Furthermore, we  analyzed prior GAO reports and Department of Homeland Security (DHS)  Office of Inspector General reports on FEMA grants administration and  catastrophic preparedness to identify any past challenges.  .
We reviewed and analyzed FEMA grant program guidance and met with  FEMA officials in the Grants Program Directorate to determine whether  other grant programs, in addition to TFER, allow for funds to be used for  catastrophic planning. We identified four major grant programs states can  use for catastrophic planning purposes: the Regional Catastrophic  Planning Grant Program, the State Homeland Security Program, the Urban  Areas Security Initiative, and the Emergency Management Performance  Grants. Other grants, such as the Port Security Grant Program (PSGP)  were considered, but discussions with officials in the Grant Programs  Directorate helped to clarify and narrow the scope of our review to the  most appropriate candidates for the types of catastrophic planning  described in the TFER grant. We collected fiscal year 2010 funding levels  for these grants, as well as information about how funds could be used for  planning, funding restrictions, and eligibility requirements.
To identify the extent to which the five states participating in FEMA’s  TFER pilot program addressed the pilot program’s stated objectives, we  analyzed information that FEMA provided in the Pilot Information  Package and further defined in its TFER Evaluation Criteria document.  The pilot program’s three stated objectives were building relationships,  developing plans, and documenting lessons learned, and FEMA provided  several criteria to measure progress towards addressing each of these  objectives. Additionally, FEMA identified key components to each  objective. We used FEMA’s criteria to assess states’ progress toward  addressing the three program objectives. Moreover, we identified  additional criteria for the building relationships objective in FEMA’s TFER  Pilot Information Package, including whether pilot states coordinated with  other states and with FEMA. These components were identified as key  components in the TFER Pilot Information Package, but were not  identified in the TFER Evaluation Criteria. Therefore, we included the  additional criteria in assessing the states’ progress toward addressing the  building relationships objective. We assigned a numerical ranking to  represent states’ progress for each objective’s components based on  interviews we conducted and our review and analysis of documents we  obtained during our site visits that included draft plans, progress reports,  and status briefings. We gathered information related to the lessoned  learned objective from state submissions to a report published by the  National Guard Bureau in August 2010, and we obtained additional lessons  learned during our interviews with state officials. Further, two of the  states provided us with the lessons learned they submitted to FEMA for its  evaluation, which is not yet complete. If the state met the criterion for a  particular component, we assigned it a ranking of ‘5’, if the state partially  met the criterion, we assigned it a ranking of ‘3’, and if the state did not  meet the criterion, we assigned it a ranking of ‘0’. Next, we averaged the  components for each of the three objectives to obtain an overall objective  score. We then assigned designations to the overall percentages of scores  as follows: averages greater than 80 percent were designated as  ‘substantial progress’; averages from 20 percent to 80 percent were  designated as ‘some progress’; and averages less than 20 percent were  designated as ‘little or no progress.’ We did not assign designations to four  criteria pertaining to developing plans and six criteria pertaining to  lessons learned because we determined that these criteria do not inform  the extent to which states met the objective and/or there was not enough  information to quantify the results.
States did not receive a designation of “substantial progress” for each  objective unless they met the criteria related to the key component of that  objective, that is, (1) developing actual plans—measured by the  completion of draft plans; and (2) building relationships with FEMA  officials, Defense Coordinating Officers, and Emergency Preparedness  Liaison Officers. FEMA did not identify key components of the third  objective, documenting lessons learned. In instances where states received  an average above 80 percent but did not meet the criteria related to these  key components, we assigned a designation of “some progress” for that  particular objective. For example, by adding the rankings assigned to  South Carolina for building relationships, the state received a score of 31  out of 35, or 89 percent. However, South Carolina did not fully meet the  criteria for coordinating with DOD and FEMA; so the state received the  designation of ‘some progress.’  Where possible, we used the Comprehensive Preparedness Guide (CPG)  101 to determine the extent to which the states’ plans adhered to FEMA’s  suggested guidelines regarding plan structure and content. About one half  of the TFER states’ 32 planning initiatives involved developing plans.  Specifically, CPG 101 recommends that basic emergency operations plans  and their annexes include the following sections: purpose, situation  overview, and planning assumptions; concept of operations; organization  and assignment of responsibilities; direction, control, and coordination;  information collection and dissemination; administration, finance, and  logistics; and authorities and references. CPG 101 provides additional  guidance for the structure of the concept of operations section for hazard- specific annexes—plans South Carolina developed—and functional  annexes—one of which Washington developed. We did not evaluate the  content of these plans because the plans remain in draft form. CPG 101  does not offer specific content guidance for the other plans types of plans  TFER states developed. We did not assign a progress designation for the  criterion that asked the extent to which states’ plans adhered to CPG 101  because all but 3 of the plans TFER states are developing remain in draft  form.
To address these objectives, we interviewed officials and obtained  information and related documents from federal agencies, the five pilot  states, and other relevant entities. Within FEMA, we met with officials  from the National Preparedness Directorate, the Grants Programs  Directorate, the Response Directorate, and officials in FEMA Regions I, III,  IV, IX, and X. Within the Department of Defense (DOD), we met with  representatives from the Office of the Assistant Secretary for Homeland  Defense and America’s Security Affairs and the Joint Chiefs of Staff. We  also met with U.S. Northern Command and U.S. Pacific Command officials  because they are the combatant commanders whose areas of  responsibilities include the United States and its territories. Further, we  met with the U.S. Department of Homeland Security, the National Guard  Bureau, the Institute for Defense Analyses, and the National Governor’s  Association. We conducted site visits to all five participant states from  June through September 2010. During our site visits, we met with the  planners hired through the TFER pilot program and other state, local,  National Guard, and DOD regional officials to discuss FEMA’s  management of the TFER pilot program and the states’ implementation of  it.
We conducted this performance audit from May 2010 to April 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Other FEMA Grants Available for Catastrophic Planning and States’ Views on Using These Grants for Planning
paragraphs: We identified four primary FEMA grant programs that states and localities  can use for catastrophic planning purposes, as shown in table 2.
Table 3 shows that these FEMA grant programs provide the states and  other eligible recipients significantly more funds than TFER provided the  pilot states. For example, in fiscal year 2010 two of the TFER pilot states  received more than $6 million from the Emergency Management  Performance Grant program compared to the $350,000 they each received  for TFER over a 24-month period. According to FEMA officials, the TFER  grant represents at most 3 percent of the grant money spent on planning  by the pilot states, as of March 2010, and all TFER activities are allowable  under other grant programs.
TFER has unique attributes from other FEMA grants. The primary  difference between TFER and these other grant programs is that the TFER  pilot states are required to use the grant funds exclusively for planning  efforts. In contrast, states may use the other FEMA grant program funds  for a full range of activities such as planning, equipment, training, and  construction and renovation. Like TFER, the Emergency Management  Performance Grants Program provides funding to states with no  requirement to pass through funding to local governments. The Regional  Catastrophic Planning Grant Program provides funds directly to urban  areas, but beginning in fiscal year 2010, states were allowed to withhold up  to 20 percent of the funds to implement or integrate the urban areas’  approved project plans throughout the state, or with neighboring states.  According to the TFER program manger, FEMA provided the TFER pilot  states with the option of continuing their efforts using Regional  Catastrophic Planning Grant Program funds; however the four states that  received funds under both programs chose not to pursue this option.  Similarly, the State Homeland Security Program requires states to pass  through at least 80 percent of the funding to local governments. Further,  the Urban Area Security Initiative provides the funding to urban areas, and  any funds retained by the state must be used to directly support the urban  area. Consequently, states cannot use most Regional Catastrophic  Planning Grant, Urban Areas Security Initiative, and State Homeland  Security Program funds to conduct statewide catastrophic planning.  However, officials in one state told us they used 2009 Emergency  Management Performance Grant funds to hire planners to continue  planning begun under TFER, and officials in two other states told us they  may use State Homeland Security Program grant funding to continue the  catastrophic planning efforts begun under TFER.
section_title: Appendix III: Task Force for Emergency Readiness (TFER) Evaluation Criteria
paragraphs: The TFER pilot program was to emphasize integration of planning efforts  across sectors, jurisdictions, and functional disciplines, as well as  integration among state, regional, and federal agencies. The Federal  Emergency Management Agency (FEMA) developed the following  questions in collaboration with the participating states to evaluate TFER.  The evaluation questions were structured around FEMA’s stated  objectives for TFER—to develop plans, build relationships, and document  lessons learned.
Our assessment of the extent to which each state addressed FEMA’s  stated objectives for the TFER pilot is presented in appendix IV. We did  not assess four criteria pertaining to developing plans and six criteria  pertaining to lessons learned because we determined that these criteria do  not inform the extent to which states met the objective and/or there was  not enough information to quantify the results. Further, we did not assess  the administrative elements of the data collection process or  administrative elements in appendix IV. Those criteria which we did not  assess are italicized below.
FEMA’s TFER Pilot Program Evaluation  1. Development of well coordinated and integrated preparedness plans  through an effective planning process.  a. Please provide a summary of the primary TFER  accomplishments.  b. Describe efforts to coordinate and integrate plans with state and  federal partners.  c. Does TFER supported planning reflect the National  Preparedness Guidelines and the full cycle of planning steps  outlined in CPG 101?  d. Is TFER planning responsive to the National Planning Scenarios  and/or other threats identified through the state’s own all-hazards  risk assessment?  e. Are the structure, organization, and content of TFER-developed  plans / annexes / appendices consistent with that recommended  in CPG 101?  f. How are TFER efforts coordinated and integrated with  resourcing efforts (e.g., training, exercises, grants, etc.)?  g. Are TFER developed plans consistent with current state  emergency plans?
Are they integrated and not duplicative?  h. Have actual plans been produced? If so, what is their impact on  state emergency readiness?  i. In what areas have TFER supported plans been developed (e.g.,  logistics, coordination, etc.)?  2. Establishment of cross-sector and interagency relationships and  protocols.  a. Does TFER supported planning reflect the formation and  involvement of the sort of integrated and collaborative planning  teams described in CPG 101?  b. Have mechanisms been established / augmented and employed  to ensure cross-sector, multijurisdictional, and interagency  planning coordination and integration?  c. Have TFER supported plans been coordinated with local  jurisdictions? Which ones? State agencies? Which ones? Federal  partners? Which ones?  d. Has a blending of TFER civil-military planning expertise been  fully exploited to assess, strengthen, and advance state  catastrophic preparedness planning?  e. How have TFER efforts incorporated the capabilities of the  private sector as partners in planning activities?  3. Documentation of lessons learned and recommendations for  improvement.  a. In the opinion of state stakeholders, has the TFER Program  resulted in demonstrable improvement in state planning  capability?  b. To what degree has state-level TFER planning been  synchronized with and improved the overall quality and  effectiveness of planning efforts?  c. How were TFER planners integrated into the state emergency  management structure and process?  d. Has the TFER Program resulted in meaningful  recommendations for the improvement of catastrophic  preparedness planning doctrine?  e. Are TFER lessons learned exportable and useful to other states  and regions to further the evolution and strengthening of their  own planning programs?  f. Has the pilot identified any additional planning / protocol /  procedure shortfalls due to integration / synchronization efforts?  g. Have any exercise After-Action Reports (AARs) or assessment  documents been developed by the site in the course of the TFER  Pilot implementation to evaluate the effectiveness of the pilot?  h. Please provide recommendations for improvements to increase  the effectiveness of the catastrophic planning programs.
FEMA also requested copies of plans developed and collected feedback on  the administrative aspects of the pilot program.  1. Data collection process. a. Please provide copies of completed / draft plans.  b. Please provide an update of the program plan, to include the  steps necessary to complete ongoing projects.
2. Administrative elements. a. Describe any challenges faced managing the TFER funds.  b. Describe the hiring and retention process for your TFER  planners. Include time lines, training, and other elements as  appropriate.  c. Describe additional administrative successes and challenges  in managing the pilot.
section_title: Appendix IV: Our Review of Each State’s Participation in the Task Force for Emergency Readiness Pilot Program
paragraphs: The Federal Emergency Management Agency (FEMA) launched the Task  Force for Emergency Readiness Pilot Program (TFER) to improve the  capacity and integration of efforts to plan a response to catastrophic  disasters. Under the TFER pilot, FEMA was to grant $350,000 to each of  five participating states—Hawaii, Massachusetts, South Carolina,  Washington, and West Virginia. FEMA designed TFER to run 18 months,  beginning on September 1, 2008, and ending on March 31, 2010, but  extended the pilot program by an additional 6 months to allow the states  more time to accomplish their goals. The TFER pilot was to emphasize  horizontal integration of planning efforts across sectors, jurisdictions, and  functional disciplines, as well as vertical integration among state, regional,  and federal agencies. For example, regarding vertical integration, states  were to integrate planning efforts with the Department of Defense (DOD)  through its Defense Coordinating Officers and Emergency Preparedness  Liaison Officers. To help develop plans to respond to catastrophic  disasters, states could hire up to three full-time planners ideally with  civilian and military planning expertise. FEMA’s stated objectives for the  TFER pilot were to build relationships, develop plans, and document  lessons learned, and states were to focus their planning on the National  Planning Scenarios, which represent examples of the gravest dangers  facing the United States, including terrorist attacks and natural disasters,  and have been accorded the highest priority for federal planning efforts.  FEMA articulated these objectives in the TFER Pilot Information Package  and the TFER Evaluation Criteria document that was ultimately to be used  to evaluate the TFER pilot. Both of these documents refer to FEMA’s  Comprehensive Preparedness Guide (CPG) 101 as a source states were to  use to guide their planning process.
In our assessment of the extent to which each state had addressed FEMA’s  stated objectives for the TFER pilot, we reviewed those documents, used  the criteria in them that we determined were measurable and appropriate  to assess the states’ progress towards meeting each objective, assigned a  numerical ranking to each objective’s criteria, and then averaged the  rankings for each objective’s criteria. Finally, we assigned designations to  these averages. When a state addressed more than 80 percent of the  criteria for an objective this was designated “substantial progress”; from  20 percent to 80 percent, “some progress”; and below 20 percent, “little or  no progress.” We gathered information related to the lessoned learned  objective from state submissions to a report published by the National  Guard Bureau in August 2010, and we obtained additional lessons learned  during our interviews with state officials. Further, two of the states  provided us with the lessons learned they submitted to FEMA for its  evaluation, which is not yet complete. We did not assign designations to  four criteria pertaining to developing plans and six criteria pertaining to  lessons learned because we determined that these criteria do not inform  the extent to which states met the objective and/or there was not enough  information to quantify the results. Four of the five pilot states are still  spending TFER funds, and the states’ status in meeting TFER’s stated  objectives may change as catastrophic planning continues. We assessed  states’ progress as of March 2011.
subsections: section_title: Hawaii
paragraphs: In figure 9, we summarize Hawaii’s TFER initiatives, note their status, and  describe the role of the planners working on them.
Hawaii is still spending TFER funds and has two TFER planners. As of  March 2011, Hawaii had made some progress in addressing FEMA’s stated  pilot program objectives. In figure 10, we depict the funding expenditure  status of the Hawaii TFER pilot and our assessment of the progress made  in the pilot program towards addressing FEMA’s stated pilot program  objectives.
Ste offici reported tht TFER plnning integrte with the te'  exiting hrricne plnning effort, which relte to  Ntionl  Plnning Scenrio.  Ste offici told us tht TFER plnner re  following the teptlined in CPG 101, but hve not completed the  fll cycle of plnning tep for it prodct, as Hii has not  completed it pl.
Hii developed plnning te tht inclded plnner from the  te civil defene diviion nd other te gencie, as well as  relevnt conty, federl, nd privte ector keholder. Ste  plnner told us they hd little contct with DOD entitie. According to  te offici, the TFER plnner’s prior militry experience was  helpfl, but not necessary for  civilin plnner to successl.
Hii offici provided recommendtion, inclding: continnd  expnd the progrm to other te, mintin flexiility to determine  work pl, nd dedicte fnding for castrophic plnning.  Addition- lly, te offici recommended tht fre castrophic plnning  progrve longer time frme thn the TFER pilot progrm,  provide more clerly defined gidnce, nd greter coordintion  etween FEMA nd DOD. Frther, fnding was not enogh to provide  competitive sarie. Hii has not exercied ny of it pl to  determine if they will e effective in the event of astrophe.
In figure 11, we detailed figure 10’s assessment of the progress made in the  Hawaii TFER pilot program towards addressing FEMA’s stated objectives  by (1) listing the criteria from the TFER Pilot Information Package and the  TFER Evaluation Criteria document used to evaluate the progress of each  objective and (2) providing our assessment score for each criterion.
We gathered additional information on Hawaii’s efforts to develop plans  and document lessons learned for the criteria we were unable to quantify.  For example, we analyzed two Hawaii draft plans, which did not contain  all of the CPG 101 components. However, we were unable to fully assess  the content of the plans because they are incomplete and work is ongoing.  Further, Hawaii officials stated that TFER allowed Hawaii to focus on  logistics planning, and without TFER, Hawaii would not be able to  undertake this planning. Hawaii reported that TFER has clearly  contributed to the development of planning task forces and has provided a  significant contribution towards state catastrophic planning. Additionally,  Hawaii officials stated that the TFER logistics planning can be used by  island territories such as Guam and Puerto Rico with similar logistical  challenges.
section_title: Massachusetts
paragraphs: In figure 12, we summarize Massachusetts’s TFER initiatives, note their  status, and describe the role of the planners working on them.
Massachusetts spent all of its TFER funds and therefore does not have any  TFER planners employed. As of March 2011, Massachusetts had made  some progress in meeting FEMA’s stated pilot program objectives. In  figure 13, we depict the funding expenditure status of the Massachusetts  TFER pilot and our assessment of the progress made in the pilot program  towards addressing FEMA’s stated pilot program objectives.
Massachusett focused on inititive pertining to rricne nd  other prioritie identified te emergency mgement offici  nd TFER plnner. The bance of it effort did not pecificlly  pertin to Ntionl Plnning Scenrio. Ste offici said the  plnner followed the teptlined in CPG 101, but cold not  complete the fll cycle of plnning ecauste offici decided tht  the Massachusett TFER plnner wold coordinte plnning effort  but pl wold e completed y other te gencie.
Rther thn developing pl, TFER plnner erved as lion to  help formte plnning te tht worked to improve the te’sastrophic prepredness. Plnner worked within the te Ntionl  Guard. According to te offici, plnning te inclded plnner  from the te emergency mgement diviion nd other te  gencie, as well as relevnt locl, federl, nd privte ector  keholder. Ste offici told us the plnners’ militry plnning  experience gve them crediility in militry circle. However, te  offici told us tht DOD entitie were minimlly involved.
Massachusett offici recommended tht FEMA continnd  expnd the progrm to other te, mintin flexiility to determine  work pl, nd dedicte fnding for castrophic plnning. Offici  ted the TFER plnner ll home level of militry experience,  which helped them cope with the logiticl chllenge nd hve  crediility with certin audience. Additionlly, offici recommended  tht fre castrophic plnning progve longer time frme  thn the TFER pilot progrnd provide more clerly defined  gidnce.  According to te offici, they hve not exercied ny  pl.
In figure 14, we detailed figure 13’s assessment of the progress made in the  Massachusetts TFER pilot program towards addressing FEMA’s stated  objectives by (1) listing the criteria from the TFER Pilot Information  Package and the TFER Evaluation Criteria document used to evaluate the  progress of each objective and (2) providing our assessment score for  each criterion.
We gathered additional information from Massachusetts on developing  plans and documenting lessons learned, but were unable to quantify this  information using FEMA’s evaluation criteria. For example, Massachusetts  TFER planners told us they often attended state and regional training and  workshops, which integrated them with resourcing efforts. However, we  did not assess progress towards meeting this objective because it is  unclear how these training and workshops inform plan development.  Further, Massachusetts officials stated that TFER was successful and  proved valuable in bringing state, local, and county agencies together with  nongovernmental organizations under the umbrella of state emergency  planning. Additionally, TFER officials stated it would have been beneficial  to have TFER planners in other New England states available to  coordinate with for events that would likely impact the entire region.
section_title: South Carolina
paragraphs: South Carolina focused its TFER pilot on planning in response to terrorist  attacks. In figure 15, we summarize South Carolina’s TFER initiatives, note  their status, and describe the role of the planners working on them.
South Carolina is still spending TFER funds and currently has two TFER  planners employed. As of March 2011, South Carolina had made some  progress in addressing FEMA’s stated pilot program objectives of building  relationships and documenting lessons learned and had made substantial  progress in developing plans. In figure 16, we depict the funding  expenditure status of the South Carolina TFER pilot program and our  assessment of the progress made in TFER towards addressing FEMA’s  stated pilot program objectives.
Sth Crolin focused on the Ntionl Plnning Scenrio relted to  terrorittck. The te has completed  reviion of it Castrophic  Incident Annex nd it has completed drft of it Terrorim Prevention  nd Repone Plnd three other pl relted to terrorittck,  which offici told us fill g in te plnning. Plnner reported they  re following the teptlined in CPG 101, but hve not finihed the  fll cycle of plnning for ny of it prodct, as Sth Crolinas not  exercied them.
Sth Crolin provided er of recommendtion, inclding:  continnd expnd the progrm, mintin flexiility to determine work  pl, nd dedicte fnding for castrophic plnning.  Additionlly,  Sth Crolin offici recommended tht fre castrophic plnning  progrve longer time frme thn the TFER pilot progrm, nd  tht FEMA hold provide more clerly defined gidnce.  Offici  identified coordinting ecrity clence cross DHS nd DOD as   chllenge thhold e reolved in imilr progr in the fre. The  ring of lesson lerned nd techniq/procedre with other  te wold hve een helpfl in developing castrophic pl  throgh the TFER pilot progrm. Sth Crolin o cited good initil  trining opportnitie throgh TFER; however thee opportnitie did  not extend throghot the life of the progrm, nd thus were not  ilable to ll Sth Crolin TFER ff.
Sth Crolin provided er of recommendtion, inclding:  continnd expnd the progrm, mintin flexiility to determine  work pl, nd dedicte fnding for castrophic plnning.  Addition- lly, Sth Crolin offici recommended tht fre castrophic  plnning progrve longer time frme thn the TFER pilot  progrm, nd tht FEMA hold provide more clerly defined  gidnce.  Offici identified coordinting ecrity clence cross  DHS nd DOD as  chllenge thhold e reolved in imilr  progr in the fre. The ring of lesson lerned nd  techniq/procedre with other te wold hve een helpfl in  developing castrophic pl throgh the TFER pilot progrm. Sth  Crolin o cited good initil trining opportnitie throgh TFER;  however, thee opportnitie did not extend throghot the life of the  progrnd thus were not ilable to ll Sth Crolin TFER ff.  Offici ted thn exercie i chedled for Mrch 2011 nd thfter-ction report will subseqently e developed.
In figure 17, we detailed figure 16’s assessment of the progress made in  South Carolina’s TFER pilot program towards addressing FEMA’s stated  objectives by (1) listing the criteria from the TFER Pilot Information  Package and the TFER Evaluation Criteria document used to evaluate the  progress of each objective and (2) providing our assessment score for  each criterion.
We gathered additional information from South Carolina on developing  plans and documenting lessons learned, but were unable to quantify this  information using FEMA’s evaluation criteria. For example, we were  unable to fully assess South Carolina’s plans because seven of eight TFER- developed plans are in draft form. However, all four of the plans state  officials provided are missing at least one of the recommended sections  described in CPG 101. State officials told us that omitted sections are  included in other state planning documents. Additionally, state officials  reported that the state used TFER to continue work on its existing work  plan, and South Carolina has developed plans and responses to some of  the threats that they would not have otherwise been able to address.
section_title: Washington
paragraphs: Washington focused its TFER pilot on developing logistics plans and on a  catastrophic incident plan. In figure 18, we summarize Washington’s TFER  initiatives, note their status, and describe the role of the planners working  on them.
Washington is spending TFER funds and as of March 2011 had one TFER  planner employed. As of March 2011, Washington had made some progress  in addressing FEMA’s stated pilot program objectives. In figure 19, we  depict the funding expenditure status of the Washington TFER pilot and  our assessment of the progress made in the pilot towards addressing  FEMA’s stated pilot objectives.
Washington elected as it primry Ntionl Plnning Scenrio astrophic erthquake nd ccompnying tsumi. Washington  integrted TFER logitic pl nd astrophic incident pln with  exiting te pl nd filled g in the te pln compendim,  ccording to te offici. The te has finlized two of it even  pl nd has finl drft for three other. Washington offici told us  they re following CPG 101 plnning process nd re trctring pln  content ccordingly.
Washington offici ted they developed plnning te tht  inclded plnner from the te Emergency Mgement Diviion  nd other te gencie, as well as relevnt locl, federl, nd privte  ector keholder. Ste offici told us the plnners’ militry  plnning experience ccelerted the lerning of civilin plnning  protocol nd helped the progrdvnce the te logitic nd  castrophic plnning. However, ccording to te offici, DOD  entitie were minimlly involved.
Washington provided recommendtion, inclding: continnd  expnd the progrm to other te, mintin the rocope of the  project eligile to ddressed throgh TFER, nd dedicte fnding  for castrophic plnning.  Additionlly, te offici recommended  tht fre castrophic plnning progrve longer time frme  thn the TFER pilot progrnd FEMA provide more gidnce.   Frther, FEMA cold improve in mgement of grnt like TFER.  Washington has exercied three ot of even of it TFER pl.
In figure 20, we detailed figure 19’s assessment of the progress made in  Washington’s TFER pilot program towards addressing FEMA’s stated  objectives by (1) listing the criteria from the TFER Pilot Information  Package and the TFER Evaluation Criteria document used to evaluate the  progress of each objective and (2) providing our assessment score for  each criterion.
We gathered additional information from Washington on developing plans  and documenting lessons learned, but were unable to quantify this  information using FEMA’s evaluation criteria. For example, Washington  selected projects for TFER which were previously identified as part of the  state’s future emergency planning work. Washington’s TFER planning also  synchronized with catastrophic planning efforts undertaken through the  Regional Catastrophic Preparedness Grant Program as both efforts  included evacuation and logistics components. However, we were unable  to assess the degree to which TFER efforts were synchronized with other  planning efforts because we did not have a basis for measuring how this  synchronization improved the effectiveness of planning efforts. Further,  Washington’s TFER plans contain some but not all of the recommended  sections described in CPG 101, but state and FEMA officials stated that  omitted sections are included in other state planning documents. We could  not fully assess the content of Washington’s plans because five out of  seven of the plans are in draft form. According to state officials,  accomplishments included drafting both catastrophic logistics plans and  developing a closer working relationship with Hawaii, Oregon, and  California by participating in Hawaii’s annual hurricane exercise.
section_title: West Virginia
paragraphs: West Virginia focused its TFER pilot on developing response plans for  three scenarios: a chemical incident, a mass evacuation of the National  Capital Region, and a dam failure. In figure 21, we summarize West  Virginia’s TFER initiatives, note their status, and describe the role of the  planners working on them.
West Virginia is still spending TFER funds and currently has one TFER  planner employed. As of March 2011, West Virginia had made some  progress in meeting FEMA’s stated pilot objectives. In figure 22, we depict  the funding expenditure status of the West Virginia TFER pilot program  and our assessment of the progress made in the pilot program towards  addressing FEMA’s stated pilot program objectives.
Wet Virgini identified three possle cenrio:  chemicl incident,  ass evuation of the Ntionl Cpitl Region, nd m filre.  Thee event cold e caused rricne or terrorittck, which  pertin to everl Ntionl Plnning Scenrio. According to te  offici, Wet Virgini' TFER plnning effort fill g in castrophic  plnning nd inclde development of astrophic incident pln.  Ste offici said tht plnner re following the teptlined in  CPG 101, but hve not finihed the fll cycle of plnning for ny of it  prodct, as Wet Virginias not completed drft pl.
Wet Virgini offici said they developed plnning te tht  inclded TFER plnner, the Deprtment of Militry Affir nd Public  Safety, the Ntionl Guard’s plnning office, other te prtner, nd  conty prtner, as well as the privte ector. Ste offici told us the  militry-civililend of plnning experience was eneficil in Wet  Virgini. However, only one of the two plnneras  militry  backgrond nd te offici told us coordintion with DOD entitieas miniml.
Wet Virgini recommended FEMA continnd expnd the progrm,  mintin flexiility to determine workpl nd dedicted fnding to  hire plnner.  Additionlly, offici recommended tht the new  progrm hve longer time frme thn the pilot nd provide more  clerly defined gidnce. Wet Virgini reported there was ignificnt  difficlty in hiring people with the reqite militry/civilin castrophic  plnning kill nd experience ecause of the hort dtion of the  progrm, which delyed the rt-p of Wet Virginia’s TFER progrm.  Wet Virginias not exercied ny of it pl.
Subsntil progress Some progress Little to no progress One of the TFER planners is a National Guardsman who was mobilized in September 2010. He is  expected to resume his TFER planning position when he returns. West Virginia hired another TFER  planner in December 2011.
In figure 23, we detailed figure 22’s assessment of the progress made in  West Virginia’s TFER pilot program towards addressing FEMA’s stated  objectives by (1) listing the criteria from the TFER Pilot Information  Package and the TFER Evaluation Criteria document used to evaluate the  progress of each objective and (2) providing our assessment score for  each criterion.
We gathered additional information on West Virginia’s efforts to develop  plans and document lessons learned for the criteria we were unable to  quantify. For example, we analyzed West Virginia’s Catastrophic Incident  Annex, which did not contain all of the CPG 101 components. West  Virginia reported that the development of this draft annex fulfilled one of  the main objectives of the pilot program. However, we were unable to fully  assess the content of the annex because it is incomplete and work is  ongoing. Additionally, West Virginia TFER planners determined during  their assessments that there were insufficient personnel available at the  county level to accomplish the tasks necessary during an incident  response. In some cases, key individuals were identified as having multiple  roles and/or responsibilities, which would be impossible to fulfill during an  actual catastrophic incident. Further, West Virginia reported the “bottom  up” approach of the West Virginia TFER pilot program is highly effective  in communicating and working with local jurisdictions, but may prove to  be less useful in an urban setting. According to state officials, West  Virginia’s TFER efforts are synchronized with the Regional Catastrophic  Preparedness Grant Program to ensure efforts are not duplicative,  particularly for the mass evacuation scenario.
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the contacts named above, the following individuals made  significant contributions to this report: Marc Schwartz and Leyla Kazaz  (Assistant Directors), Courtney Reid, Susanna Kuebler, Paul Hobart, Jena  Whitley, Katrina Moss and Cody Loew; Mae Jones, Lara Miklovek, Amie  Steele, Cynthia Saunders, Kate Lenane, and Tracey King.
section_title: Related GAO Products
paragraphs: Disaster Response: Criteria for Developing and Validating Effective  Response Plans. GAO-10-969T. Washington, D.C.: September 22, 2010.
Homeland Defense: DOD Can Enhance Efforts to Identify Capabilities to  Support Civil Authorities During Disasters. GAO-10-386. Washington,  D.C.: March 30, 2010.
Homeland Defense: DOD Needs to Take Actions to Enhance Interagency  Coordination for Its Homeland Defense and Civil Support Missions.  GAO-10-364. Washington, D.C.: March. 30, 2010.
Interagency Collaboration: Key Issues for Congressional Oversight of  National Security Strategies, Organizations, Workforce, and  Information Sharing. GAO-09-904SP. Washington, D.C.: September 25,  2009.
Homeland Defense: U.S. Northern Command Has a Strong Exercise  Program, but Involvement of Interagency Partners and States Can Be  Improved. GAO-09-849. Washington, D.C.: September 9, 2009.
National Preparedness: FEMA Has Made Progress, but Needs to  Complete and Integrate Planning, Exercise, and Assessment Efforts.  GAO-09-369.Washington, D.C.: April 30, 2009.
Emergency Management: Observations on DHS’s Preparedness for  Catastrophic Disasters. GAO-08-868T. Washington, D.C.: June 11, 2008.
National Response Framework: FEMA Needs Policies and Procedures to  Better Integrate Non-Federal Stakeholders in the Revision Process.  GAO-08-768. Washington, D.C.: June 11, 2008.