Federal Emergency Management Agency: Strengthening Regional Coordination Could Enhance Preparedness Efforts
Published: 2016-02-04T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DHS's FEMA is responsible for coordinating with state, local, and tribal governments to prepare for disasters. Specifically, FEMA provides preparedness grants to states and localities, and works to implement the National Incident Management System nationwide, among other things.
GAO was asked to review FEMA's efforts to enhance coordination for regional preparedness. This report addresses the extent to which FEMA and regional offices have (1) addressed preparedness grant management coordination challenges, (2) established a system to assess NIMS implementation, and (3) collaborated with RAC stakeholders. GAO analyzed FEMA documentation on grant management, NIMS evaluation, and RACs. GAO surveyed RAC members, visited 4 regional offices selected for their geographic representation, spoke with state emergency management officials from 10 states, and interviewed FEMA officials. The site visits cannot be generalized but provided insights.
section_title: What GAO Found
paragraphs: GAO found that the Department of Homeland Security's (DHS) Federal Emergency Management Agency (FEMA) has taken some steps, but has not fully addressed, preparedness grant management coordination challenges between headquarters and its regions. For several preparedness grant programs, FEMA headquarters and regions share management and monitoring responsibilities. Assessments by FEMA and others since 2009 have recommended that FEMA regions manage and monitor preparedness grants to avoid confusion and duplication and strengthen coordination with state and local grantees. However, in 2012 FEMA changed course and decided to continue sharing grant management between headquarters and regions. Since then, FEMA officials said they are taking steps to address coordination challenges that exist in this structure. However, GAO found that challenges continue to exist. For example, states and FEMA regional officials told GAO that FEMA headquarters and regions did not always coordinate monitoring visits and provided inconsistent guidance to grantees. Further, while FEMA officials identified some steps to address the challenges, FEMA lacks a plan with time frames and goals for addressing them. Doing so will better enable FEMA to effectively address the long-standing challenges in managing preparedness grants.
FEMA uses states' self-assessments to determine if states have implemented the National Incident Management System (NIMS), as required, but does not assess NIMS implementation using the results of preparedness exercises. NIMS is a comprehensive, national approach to incident management and provides a framework to enable all levels of government and the private sector to work together to prepare for, prevent, respond to, and recover from incidents. Although states generally report high levels of NIMS implementation, officials from all four FEMA regional offices and 9 of 10 states GAO spoke with said that the NIMS self-assessments are perfunctory and do not measure whether, or how well, NIMS is being implemented. FEMA officials said they do not verify states' self-reported NIMS implementation information because of the scope and breadth of the information. All of the FEMA regions and 8 states said the best way to assess NIMS implementation is to review states' performance in after-action reports following exercises and real-world events. However FEMA officials do not review these reports to assess NIMS implementation. Doing so could allow FEMA to better assess NIMS implementation and identify areas for improvement.
All 10 FEMA regions established Regional Advisory Councils (RAC) to provide advice on emergency management issues specific to the region, and about 90 percent of RAC members reported in a GAO survey that meetings are useful for collaborating with their FEMA regional office. However, some regions do not routinely meet with their RACs, and some do not consistently report back to RAC members on the status of their recommendations to FEMA. By more routinely obtaining input from RAC members and by providing timely feedback on recommendations, FEMA regional offices could better ensure they are coordinating with key stakeholders and identifying areas for strengthening preparedness.
section_title: What GAO Recommends
paragraphs: GAO recommended, among other things, that FEMA develop a plan for addressing long-standing coordination challenges with grant management, review after action reports to assess NIMS implementation, and improve coordination with RACs. Although DHS did not concur with the grants management recommendation, GAO continues to believe challenges documented in the report support the recommendation. DHS concurred with the other 4 recommendations and described the actions they planned to take in response.

section_title: Letter
paragraphs: The Federal Emergency Management Agency (FEMA), within the  Department of Homeland Security (DHS), through its 10 regional offices  is responsible for coordinating government-wide disaster response efforts  and delivery of all of FEMA’s programs and activities to state, tribal, and  local partners; nongovernmental organizations; and citizens across the 56  states and territories. According to FEMA, disaster response begins and  ends with the affected FEMA regional office in the lead, and regional  FEMA personnel have central roles throughout the life cycle of an  incident. We previously reported how regional coordination efforts of  state and local stakeholders can enhance preparedness.
FEMA regions coordinate with state and local stakeholders in various  ways in order to strengthen national preparedness to prevent and  respond to domestic terrorist attacks, major disasters, and other  emergencies. For example, from fiscal years 2002 through 2015, DHS  awarded over $40 billion to state, local, tribal and territorial grant  recipients in preparedness grants to strengthen national preparedness  capabilities, and FEMA’s regional offices are responsible for financial  management and, for selected programs, programmatic management of  these grants. The regional offices also work directly with states, tribes,  and territories to implement the National Incident Management System  (NIMS), a standardized approach to guide emergency responders at all  levels of government and the private sector to coordinate efforts to  respond to incidents and save lives and property. All states and  territories must agree to adopt and implement NIMS and certify their  compliance with NIMS as a requirement to receive preparedness grant  funding from FEMA. To enhance coordination with regional stakeholders,  regional offices convene Regional Advisory Councils (RAC) to obtain  insight into emergency management issues at the state and local levels.
The Post-Katrina Emergency Management Reform Act of 2006 (Post- Katrina Act) was enacted to address various shortcomings in the national  response, including federal, state, and local preparedness capabilities,  identified in FEMA’s preparation for and response to Hurricane Katrina.  The Post-Katrina Act included several requirements that address FEMA’s  regional coordination with state, local, and tribal governments. For  example, the Post-Katrina Act called for the establishment of regional  offices that would work with state, local, and tribal governments to ensure  effective, coordinated, and integrated regional preparedness, among  other things. As part of this effort and to address grant management  coordination challenges between headquarters and the regions, FEMA  developed plans to transfer grant management functions from  headquarters to the FEMA regions (regionalization). The Post-Katrina Act  also mandated that FEMA establish a comprehensive system to assess  compliance with NIMS, among other things, and required that each  FEMA regional office establish a RAC with members from state, local and  tribal stakeholders to advise regional administrators in each of FEMA’s 10  regional offices on emergency management issues specific to their  regions.
You requested that we review FEMA’s efforts to enhance coordination for  regional preparedness. This report assesses the extent to which FEMA  headquarters and regional offices have (1) addressed grant management  coordination challenges between headquarters and the regions, (2)  established a comprehensive system to assess NIMS implementation,  and (3) collaborated with RAC members.
To address the first objective, we gathered and reviewed relevant  documentation, such as FEMA assessments of grant management by the  Grant Programs Directorate (GPD), FEMA’s previous delegations of grant  management responsibilities to regional offices, agency task force reports  on advantages and disadvantages of regionalization, FEMA headquarters  and regional office grant management roles and responsibilities, FEMA  documentation on regionalization pilot programs, and other memoranda  and internal documents. We also reviewed prior GAO reports on regional  preparedness, grant management, and consolidation of management  functions. In addition, we interviewed FEMA’s Director of the  Preparedness Grants Division within GPD and other senior GPD officials  to discuss programmatic grant management as well as previous FEMA  assessments of the impact of moving management responsibilities to  FEMA regional offices. We also interviewed grant management officials  from 4 FEMA regional offices to discuss financial grant management and  coordination of monitoring activities with GPD. These offices were  selected in order to provide a mix of geographic locations and whether  the regional office had been selected to participate in FEMA grant  management pilot programs. Similarly, we interviewed officials from 9  states in the 4 FEMA regions to discuss grant management within the  states as well as interactions with FEMA grant management officials.  These states were selected to reflect a diversity of experiences from  FEMA regions we chose. While the information gained from these  interviews cannot be generalized across all states, it provides useful  insights into the nature of FEMA GPD and regional coordination with  states with regard to grant management.
To address the second objective, we reviewed NIMS-related documents  such as the NIMS Doctrine, NIMS Compliance Guidance, NIMS  implementation reports, and results of the NIMS implementation  questions in the Unified Reporting Tool (URT). We also reviewed  documents on the Homeland Security Exercise and Evaluation Program  (HSEEP) and on preparedness grants requirements. We compared  FEMA’s efforts in NIMS implementation and verification with Homeland  Security Presidential Directive-5 (HSPD-5) and with language in OMB’s  Circular No. A-11, regarding data validation and verification. We  interviewed officials at FEMA’s National Preparedness Directorate, which  includes the National Integration Center (NIC)—the office responsible for  coordinating and enabling NIMS implementation to discuss the NIC’s  efforts in implementing NIMS. To discuss NIMS implementation efforts at  the regional and state levels, interactions between regional and state  officials, and mechanisms for verifying how well NIMS is being  implemented, we also interviewed officials from the 4 FEMA regional  offices and 9 states mentioned above. In addition, because of the  recommendation of officials in several regional offices and states, we  interviewed an official from a 10th state from within the selected FEMA  regions. While the information gained from these interviews cannot be  generalized across all regions and states, it provided useful insights into  the various efforts in implementing NIMS. Additionally, we analyzed all 35  full-scale exercise after-action reports (AAR) that FEMA received for  exercises conducted during fiscal year 2014 in the 4 regions we visited.  The purpose of the analysis was to determine if they could be used to  assess the level of participants’ implementation of NIMS. While the  results of these exercises are not generalizable to the country as a whole,  they do provide useful insights into their possible use to assess NIMS  implementation.
To address the third objective—to assess regional offices’ collaboration  with RAC members—we interviewed officials at FEMA headquarters with  responsibilities for regional operations and regional office officials from  the 4 regions we visited. We also reviewed FEMA and DHS Office of  Inspector General (OIG) reports with information and findings relevant to  RACs. In addition, we contacted FEMA RAC liaisons from all 10 regions  by e-mail to obtain information about their RACs and RAC members. We  attended two RAC meetings in two regions that took place during the  early part of our review, and obtained documentation of agendas and  other relevant materials for the RACs in all 10 FEMA regions. In addition,  we conducted a self-administered web-based survey from May 27  through June 29, 2015 of 110 RAC members FEMA identified as active  from all 10 regions and 77 responded, for a response rate of 70 percent.  For further information on our survey, see appendix I; for the survey  questions and results, see appendix II. We also used information  obtained from interviews with FEMA headquarters and regional office  officials involved in communicating and collaborating with RACs. Further  details on our objectives, scope, and methodology are contained in  appendix I.
We conducted this performance audit from October 2014 to February  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Preparedness Grant Management
paragraphs: FEMA preparedness grant management is composed of different financial  and programmatic management functions. FEMA created GPD in April  2007, to consolidate oversight of all FEMA grants. GPD, in FEMA  headquarters, provides subject matter expertise in response to regional  office and stakeholder inquiries; develops grant guidance; and delivers  policy, training, and systems and data analysis. The 10 FEMA regions are  responsible for financial monitoring, implementing corrective action plans,  as well as other follow-up activities with grantees. Financial management  functions include monitoring grantee expenditures, amending financial  terms of grants, and closing out grants. Program management includes  developing grant application packages creating program standards, and  monitoring grantee activities to ensure alignment with homeland security  strategies.
FEMA’s regional offices have program management responsibility for  various legacy programs primarily because they managed those  programs prior to the establishment of GPD. The largest of these legacy  programs is the Emergency Management Performance Grant (EMPG),  which provided over $350 million in fiscal year 2015 to assist local, tribal,  territorial, and state governments in enhancing and sustaining all-hazards  emergency management capabilities. GPD has management  responsibility for the Homeland Security Grant Program, which provides  more than $1 billion annually to help strengthen the nation against risks  associated with acts of terrorism and other catastrophic events.
section_title: National Incident Management System
paragraphs: FEMA works with state and local stakeholders to implement NIMS, a  comprehensive, national approach to incident management that is  applicable at all jurisdictional levels and across functional disciplines and  the full spectrum of potential incidents, hazards, and impacts, regardless  of size, location, or complexity. DHS established NIMS in 2004 to improve  coordination and cooperation between public and private entities in a  variety of incident management activities and provide a common standard  for overall incident management. NIMS implementation consists of  training, using a standardized way of defining and categorizing  emergency resources known as resource typing, and incorporating a  standardized organizational emergency response structure called the  Incident Command System into emergency management plans, policies  and procedures, among other things. States, tribes, and local  jurisdictions are required to adopt and self-assess their level of NIMS  implementation, and certify their compliance and report it to FEMA  through the URT in order to receive preparedness grant funding from  FEMA. Additionally, states receiving preparedness grants are required  to develop and maintain an exercise program consistent to the degree  practical with HSEEP in support of the National Exercise Program.  HSEEP exercises identify exercise objectives and align them to FEMA’s  32 “core capabilities” for evaluation during the exercise. In order to report  on the required exercises, states receiving preparedness grants are  required to prepare after-action reports—evaluations of performance—on  core capabilities assessed in the exercise, and to submit those to FEMA.
FEMA established the NIC in 2004. The NIC, a division of FEMA’s  National Preparedness Directorate, is responsible for the ongoing  management and maintenance of NIMS, including developing guidance  to define and categorize the resources and job position qualifications  requested, deployed, and used in incidents. Specifically, the NIC is to  coordinate with state and local stakeholders to develop national  standards, guidelines, and protocols for incident management.
In 2008, FEMA established new regional positions for federal  preparedness coordinators (FPC), who direct and coordinate the activities  of each regional national preparedness division to ensure regional  implementation of national preparedness programs, policies, goals, and  objectives. FPCs are to evaluate preparedness program activities to  assess their effectiveness; changes in risk or capability; and the  performance of preparedness program activities including after-action  reports, training evaluations, lessons learned, and corrective actions. For  example, FPCs are responsible for monitoring NIMS compliance and  implementation, in addition to other national preparedness initiatives.  Further, FPCs are to proactively engage stakeholders and acquire an  understanding of their preparedness efforts and also assist in the  planning, design, execution, and evaluation of federal, state, local, and  regional exercises. FEMA regional NIMS coordinators act as subject  matter experts regarding NIMS for the local, state, territorial, and tribal  nation governments within their FEMA regions, as well as for the FEMA  regional administrator and staff.
section_title: Regional Advisory Councils
paragraphs: The Post-Katrina Act requires that each FEMA regional administrator  establish a RAC with members from state, local, and tribal entities to  provide advice on emergency management issues specific to the  region. While the Post-Katrina Act does not specify how many members  a RAC should have, the law states that a state, local, or tribal government  located within the geographic area served by the regional office may  nominate officials, including emergency managers, to serve as members.  Each of the 10 FEMA regional offices established a RAC after the Post- Katrina Act was enacted in 2006. In August 2011, FEMA developed a  RAC charter template that each of the regional offices voluntarily  adopted; the charters require that RACs meet twice annually in order to  provide the types of advice on emergency preparedness in their regions  to the FEMA regional administrators listed in the Post-Katrina Act.
In addition to the RACs, the Post-Katrina Act also required that FEMA  establish the National Advisory Council (NAC) to ensure effective and  ongoing coordination of federal preparedness efforts, among other  things. The NAC advises the FEMA Administrator on all aspects of  emergency management, and incorporates state, local, and tribal  governments; nonprofit; and private sector input in the development and  revision of the National Preparedness Goal, the National Preparedness  System, the National Incident Management System, and other related  plans and strategies.
FEMA’s 10 regional offices are the principal conduit for delivery of  preparedness programs and activities to state, tribal, and local partners;  non-governmental organizations; the private sector; and citizens. Such  preparedness activities include, among other things, stakeholder  coordination and information sharing, consulting, planning support,  capability assessments and reporting, exercise performance and  evaluation, and internal and external training. Figure 1 shows FEMA’s 10  regions and the states and territories that compose each region (the  District of Columbia is in Region III).
section_title: GPD Lacks a Plan to Address Long- standing Challenges in Coordinating Preparedness Grants between FEMA Headquarters and Regions
subsections: section_title: Past Assessments Supported Moving Grant Management Functions to the Regions to Address Oversight and Coordination Problems, but FEMA Decided Not to Change Its Approach
paragraphs: Currently FEMA uses a hybrid management model for preparedness  grant programs, with shared responsibilities between headquarters and  the regions. In October 2009, in response to a congressional request,  the National Academy of Public Administration (NAPA) conducted a study  and recommended that FEMA transfer all preparedness grant authorities  to its regional offices. Similarly, in May 2010, FEMA’s Office of Policy  and Program Analysis (OPPA) reported that FEMA’s hybrid grants  management model created significant inconsistencies in managing  preparedness grants, and recommended that programmatic management  functions for preparedness grants should be transferred to the regional  offices. In September 2010, GPD’s Regional Implementation of Grants  (RIG) task force also reported that programmatic grant management  functions for grants should be moved to the 10 regional offices, because  of considerations of customer service, efficiency, people, processes, and  systems. In July 2011, we reported that GPD’s efforts to regionalize  management responsibilities for preparedness grants were consistent  with internal control standards. On the basis of the results of our review  of FEMA’s plans and efforts to regionalize grant management functions,  we did not make recommendations at that time.
FEMA’s Deputy Administrator for Protection and National Preparedness,  in consultation with FEMA’s Administrator, decided in early 2012 against  pursuing further regionalization of grant management functions, according  to FEMA’s Assistant Administrator for Grant Programs. The Assistant  Administrator said their decision was based on, among other things,  estimates that the costs of regionalization would be greater than the  annual savings identified in the earlier OPPA study and FEMA  management’s belief that risks associated with the change, such as  inconsistent program implementation across the regions, outweighed the  potential benefits. As a result, FEMA continues to use its hybrid  management model for preparedness grant programs, with shared  responsibilities between headquarters and the regions.
section_title: FEMA Regional Offices and Grantees Identified Challenges with FEMA’s Hybrid Management Model
paragraphs: Long-standing challenges associated with the separation of management  functions under the hybrid model of grant management, including the lack  of coordination of grant-monitoring visits, and coordination and  implementation of guidance provided by headquarters officials to state  officials, continue to create challenges.
In 2010, GPD’s RIG task force report cited examples of how  programmatic and financial monitoring visits, conducted by FEMA GPD  staff and regional offices respectively, occurred independently of one  another and were disruptive to the state emergency management  agency’s day-to-day operations. In response to the lack of coordination  in monitoring site visits, GPD’s Deputy Assistant Administrator sent a  memorandum to staff in 2011 citing concerns that it had become  “apparent GPD HQ staff continue to do program monitoring visits without  proper notification to the regional offices” and directing that headquarters  personnel conduct all monitoring visits in conjunction with regional staff,  changing the dates of any planned headquarters visits to accommodate  regional staff schedules. While the memorandum called for disciplinary  action against any GPD personnel who did not follow the directive, GPD  officials did not identify any instances where disciplinary action had been  taken.
We also found challenges in the coordination of site visits. Officials in 3 of  9 states we talked with on grants issues stated that monitoring visits from  FEMA GPD and the regional offices do not appear to be well coordinated.  For example, officials in 1 state said that in September 2014, GPD and  regional grants staff conducted site visits during the same week; however,  neither the GPD staff nor the regional staff knew that the other was  conducting a monitoring visit that week. The state officials stated that they  had to leave one monitoring meeting early to go to another monitoring  meeting by a different group of grant management staff. Similarly,  regional grants officials in 2 of the 4 regions we visited said in 2015 that  they may or may not receive information on monitoring activities  undertaken by GPD officials, depending on who is conducting the  monitoring activities. A grant official in 1 region we visited stated that GPD  conducted a site monitoring visit as recently as September 2015, but did  not inform regional staff of this visit. Regional grants management officials  in all 4 regions we visited stated that not having the results of  programmatic monitoring creates challenges in financial monitoring. For  example, financial monitoring may identify a significant drawdown of grant  funds; however, without program information, regions cannot determine if  this corresponds to significant progress in program implementation.
Further, challenges related to coordination and implementation of  guidance provided by GPD (headquarters) officials, FEMA regional  officials, and state officials has been a long-standing problem. For  example, in 2009, NAPA identified regional challenges associated with  assessment activities including grant monitoring, noting that FEMA  regional and headquarters units “need to reduce burden on states and  ensure assessments-requirements are coordinated among FEMA  divisions and the federal sector” and that new documents from FEMA  headquarters “typically have a short review and comment period, and  adherence is often required prior to finalization.” Similarly, the May 2010  RIG Task Force Report noted that “the timeliness of communication and  information distribution with the grantee community needs to be  improved” and the need to “formally establish regional liaisons to support  the consistent prioritization of communication within GPD and with  grantees.”
We found that these issues related to coordinating guidance provided by  headquarters officials to state officials continue to create challenges.  Officials from all 4 FEMA regions we visited said that GPD officials had  provided guidance to state grantees that contradicted guidance given by  the regional staff. Further, officials in 2 of the 4 regional offices we visited  stated that incorrect guidance given by GPD officials to state grantees  resulted in deficiencies during financial monitoring conducted by regional  grants personnel. Specifically, according to FEMA grant management  officials in 1 region, a grantee, at the advice of GPD officials, shifted funds  from one grant program into other grant programs, but regional grants  personnel later found, as part of their financial monitoring, that this was a  violation of regulations and required that the funding be returned.  Similarly, FEMA grant officials in another region stated that a new grantee  that received transit grants drew down the entire grant amount soon after  the grant was awarded, which is a violation of FEMA grant management  regulations. Regional officials asked the grantee to return the funds;  however, the grantee stated that GPD officials had told him he could draw  down these funds. Regional officials added that had the grantee not  returned the funds, the next step would have been to do an official  collection; however, regional offices do not have the authority to do this  because the grant program is managed from FEMA headquarters by  GPD. Officials from 3 of the 9 states we visited told us that they had  received contradictory information from regional staff and GPD staff  regarding the management of grant programs.
GPD officials agreed that there were continuing opportunities to improve  coordination and communications in their management of preparedness  grants and said that FEMA had taken steps during the course of our  review to facilitate coordination and communication. For example, GPD  officials said that they had taken steps to improve coordination of site  visits and that, beginning in fiscal year 2013, GPD made integrated  financial and programmatic monitoring a priority, and incorporated this  into its monitoring plans. According to FEMA’s fiscal year 2014 grants  monitoring plan, GPD officials are to contact regional grant staff when  monitoring activities are initiated and document this interaction. GPD  officials did not provide any documentation that such coordination had  occurred. Nonetheless, they identified two joint GPD and regional office  monitoring visits that took place during the summer of 2015 (at the  request of the regions).
GPD officials also noted that, in a February 2015 meeting with regional  officials, GPD and the regional offices agreed to the following: elevate grant-related issues to regional administrators and to the  Assistant Administrator for Grant Programs, provide regional grants staff visibility on programmatic grant reviews  performed by GPD staff,  conduct desk reviews in coordination with the regions and provide  copies of the final reports to the regional offices, provide GPD’s annual monitoring plan to the regional offices and  conduct joint monitoring of the same states for programmatic and  financial issues when possible, provide quarterly analytical reports of grant-monitoring-related issues  to all regional offices, and  schedule regular calls between GPD management and regional  grants division directors.
These most recent efforts to improve coordination of preparedness grant  management functions are not significantly different from the historical  efforts GPD has taken. For example, GPD cited monthly coordination  calls between GPD and the regional offices as a coordinating effort meant  to address challenges we identified. However, FEMA also cited these  calls as a coordination mechanism in 2012 and noted that these calls had  been a recurring effort since 2008. Similarly, providing copies of GPD’s  annual monitoring plan and copies of reviews performed by GPD do not  appear to have resolved the underlying challenges associated with GPD’s  hybrid grants management model.
GPD officials also said they intend to look at FEMA’s overall grants  monitoring efforts to see what can be done to improve the program for  fiscal years 2016 and beyond, including risk assessments, general  assessments, on-site desk reviews, reports with real-time data and  metrics, and corrective action plans. However, FEMA has not developed  a plan with time frames, goals, metrics, or milestones for how and when it  will address long-standing coordination challenges associated with the  existing hybrid grants management model identified in previous FEMA  assessments and in our review. According to FEMA’s Assistant  Administrator, GPD has involved the regions in performance measure  development since September 2011 and has communicated the  measures and results to the regions. These performance measures  include, among other things, percentage of preparedness grant funds  monitored and percentage of grants award determined to be low risk.  However, these performance measures do not address the deficiencies in  coordination identified in our review.
According to effective program management practices, specific goals and  objectives should be conceptualized, defined, and documented in the  planning process, along with the appropriate steps, time frames, and  milestones needed to achieve those results. Given the longstanding  coordination challenges, establishing a plan with time frames, goals,  metrics, and milestones could help FEMA headquarters and regional  offices address these challenges.
section_title: FEMA Could Have Better Visibility into NIMS Implementation by Using the Results of Preparedness Exercises
subsections: section_title: The NIC Relies on States’ Self-assessments to Determine NIMS Implementation
paragraphs: The NIC uses a self-assessment tool to determine if all states and  territories have adopted and implemented NIMS elements into their  emergency response planning and training. The NIC relies on a self- assessment tool, as the Post-Katrina Act requires that FEMA develop an  assessment system to determine compliance with NIMS but does not  require that FEMA physically verify compliance. Since 2005, all 56 states  and territories have been required to meet NIMS compliance and  implementation requirements to be eligible to receive federal  preparedness grants. The NIC began using the latest version of its tool  for states to submit self-assessments—the URT—in 2013. In the URT  spreadsheet, states and territories fill in responses to a series of  questions aimed at determining the level of NIMS implementation.  Generally, states and territories gather this information from their  subjurisdictions and consolidate the information at a state level to report  to FEMA. (For a list of questions, see app. III.)
Although states generally report high levels of NIMS implementation in  their URTs, officials from all four FEMA regional offices we spoke with  and from 9 of the 10 states we spoke with said that the NIMS self- assessments are perfunctory and do not necessarily measure whether, or  how well, NIMS is being implemented. For example, in fiscal year 2013,  100 percent of the states and territories reported having “formally  adopted” NIMS, and incorporated NIMS principles and concepts into  training and exercises. Similarly, in fiscal year 2014, states and  territories reported that 93 percent of their subjurisdictions had formally  adopted NIMS, about 91 percent had incorporated NIMS concepts and  principles into training, and about 93 percent incorporated NIMS into  exercises. FEMA regional and state officials we spoke with said there is  an incentive to report compliance with NIMS regardless of the actual  status of implementation since it is a requirement for receiving  preparedness grants. According to preparedness grant guidance, there is  no requirement for states to verify NIMS compliance or implementation at  the subjurisdiction level. However, of the states we spoke with on NIMS  implementation, 1 had implemented its own NIMS compliance check  program. Specifically, a state emergency management official in 1 state  we spoke with said that his state has initiated its own reviews of NIMS  implementation and found that some reporting of full NIMS  implementation by subjurisdictions were questionable. For example, in  one subjurisdiction, state officials found the following:
The subjurisdiction could not produce training records to show that  emergency management personnel had received NIMS training.
The requirement to incorporate NIMS principles into the  subjurisdiction’s emergency operations plans had not been  completed.
There were no records of exercises or corrective action plans  available to identify if NIMS had been incorporated into the  subjurisdiction’s exercise program.
While this particular example cannot be generalized to other states, it  shows that in some cases, subjurisdictions within one state with a NIMS  compliance check program are reporting levels of NIMS implementation  that may not fully reflect the status of local NIMS efforts.
section_title: Existing Assessments after Exercises and Real- World Events Provide a Means of Further Assessing States’ Self- reporting on NIMS Implementation
paragraphs: Officials from all 4 regional offices and from 8 of the 10 states we spoke  with said that the best way to assess how well NIMS has been  implemented is by assessing states’ performance in preparedness  exercises and real-world events. They all agreed that if NIMS principles  have been well implemented, responses to disasters will be more  effective because the objective of NIMS is to ensure that all levels of  government across the nation work together efficiently and effectively  using a single, national approach to an incident. Further, according to the  NIMS doctrine, using a comprehensive national approach improves the  effectiveness of emergency management personnel’s responses to the  full spectrum of potential incidents and hazard scenarios, including  natural hazards, terrorist activities, and other manmade disasters. To  capture information on states’ performance during preparedness  exercises and real-world events, states generally develop after-action  reports. The EMPG grant program, under which all states and territories  receive funding, specifically requires participants to develop and maintain  an exercise program, consistent to the degree practical with HSEEP in  support of the National Exercise Program, and to conduct preparedness  exercises and submit AARs that evaluate the state or territory’s  performance during the exercises based on one or more core  capabilities. Under HSEEP—which provides a common approach to  designing, developing, conducting, and evaluating exercises—planners  identify exercise objectives and align them to core capabilities. In some  cases, the exercise objectives test one or more of the NIMS elements  identified by FEMA that should be incorporated into training. As such,  exercises could also allow for identifying areas of needed improvement in  NIMS implementation, one of the legislative bases for establishing the  National Exercise Program.
FEMA regional officials we spoke with stated that some of FEMA’s  national core capabilities are good proxies for how well NIMS is being  implemented, and that AARs—which states submit to the National  Exercise Division—can provide insight into the level of NIMS  implementation when certain of these core capabilities are assessed.  NIC officials stated if an exercise objective includes testing an element of  NIMS, then it is possible to assess NIMS implementation. For example,  evaluating a state’s performance related to the core capability of  operational coordination could include as an exercise objective to  evaluate the command and control of an incident according to NIMS  principles. Similarly, the core capability for operational communications  could provide an assessment related to the NIMS elements for  interoperability of compatible communications, technology, and  information management.
We analyzed AARs for 35 full-scale exercises that were conducted in  fiscal year 2014 in 4 FEMA regions, 33 of which assessed at least one, if  not two or all three, of the core capabilities we identified as proxies for  NIMS elements—operational coordination, operational communication,  and public information and warning. Of the 35 reports we reviewed, 30  assessed operational coordination, 22 assessed operational  communication, and 12 assessed public information and warning.  Generally, for the capabilities assessed, the AARs identified strengths as  well as issues and challenges in performing related activities and tasks.  Specifically, 24 of the 30 exercises that assessed operational  coordination, 21 of the 22 exercises that assessed operational  communication, and 8 of the 12 exercises that assessed public  information and warning identified areas for improvement with those  capabilities. While exercises are meant to assess the ability to meet  exercise objectives by documenting strengths, areas for improvement,  core capability performance, and corrective actions in an AAR, some of  the AARs documented problems with NIMS-related exercise objectives,  indicating that there is room for improvement in NIMS implementation.  For example, four of the AARs found that an Incident Command System  (ICS) was not established among responding agencies, even though  NIMS calls for using such a structure. Similarly, another AAR concluded  that responding staff were unfamiliar with their position-specific roles and  responsibilities, even though NIMS calls for emergency management and  response personnel to have a clear understanding of their roles and  responsibilities. The report noted that the staff displayed difficulty in  monitoring, gathering, receiving, organizing, and sharing incident  information—all activities that implementation of NIMS is designed to  facilitate. Another exercise evaluating operational communication found  that there was no communication among the different law enforcement  agencies and officers during the exercise. These examples demonstrate  that AARs can provide a basis for FEMA to develop a more  comprehensive and sophisticated understanding of states’ NIMS  implementation, and suggests that there is room for improvement in NIMS  implementation for some jurisdictions.
NIC officials said they do not verify the self-reported NIMS  implementation information that states submit because of the scope and  breadth of the information. However, in reviewing AARs to help inform  FEMA’s understanding of NIMS implementation, FEMA regional office  staff could be in a better position to assess how well states have actually  implemented NIMS, as regional staff have more direct responsibilities  related to states’ NIMS implementation. Specifically, FPCs—who lead  regional efforts to implement the National Preparedness System across  the federal, state, tribal, and local jurisdictional levels—are responsible for  monitoring NIMS compliance and implementation. FPCs are also  responsible for assisting in planning, design, execution, and evaluation of  exercises. In addition, each regional office has a NIMS coordinator who is  responsible for working with states to provide NIMS information and  technical assistance to support, facilitate, and enhance their  implementation of NIMS. One FPC stated that officials in his region  review state AARs within the region to identify needed corrective actions  that could be taken by states. Similarly, one regional NIMS coordinator  developed a checklist to perform NIMS implementation state monitoring  visits and said he visits 2 states each year in his region to ask questions  about implementation of various NIMS principles and provide assistance.  However, while some of these officials have taken independent steps to  assess some level of NIMS implementation, the regional offices we spoke  with do not systematically assess states’ level of NIMS implementation  by, for example, reviewing AARs because there are no policies or  procedures that call for these activities. Specifically, while states are to  submit AARs to FEMA’s National Exercises Division, and to the regional  EMPG Program Manager, as a requirement for receiving EMPG grants,  FPCs and NIMS coordinators are not required to review them to provide  validation of the NIC’s NIMS implementation assessment.
FEMA regional office officials we spoke with, along with several state  emergency management officials, agreed that assessing NIMS  implementation using AARs could provide FEMA a way to further assess  and elaborate on the information states and territories currently provide.  For example, the National Exercise Division, another division within the  National Preparedness Directorate, conducted an assessment of NIMS in  2013 by reviewing AARs and found that failures in NIMS performance  were predominantly due to failures in implementation of NIMS. The report  recommended, among other things, that the agency modify existing  training and exercises to allow responders to integrate NIMS concepts  and improve assessments of the ICS (as a component of NIMS  performance) by incorporating relevant principles as criteria of  performance. According to a NIC official, FEMA has been working on  the recommendations and expects to implement the actions when FEMA  releases an update to the NIMS doctrine. Finally, while the National  Exercises Division did assess NIMS implementation issues in 2013 by  analyzing AARs, regional offices and the NIC do not systematically or  regularly assess NIMS implementation. Using AARs to systematically  assess NIMS implementation over time could allow the regions and the  NIC to identify recurring issues in jurisdictions.
HSPD-5 calls for FEMA to (1) establish a mechanism for ensuring  ongoing management and maintenance of the NIMS, including regular  consultation with other federal departments and agencies and with state  and local governments, and (2) develop standards and guidelines for  determining whether a state or local entity has adopted NIMS. In addition,  OMB Circular No. A-11 states that data limitations can lead to bad  decisions resulting in lower performance or inaccurate performance  assessments, and data limitations can include imprecise measurement  and recordings, incomplete data, and inconsistencies in data collection  procedures. Consistent verification and validation of performance data  support the general accuracy and reliability of performance information,  reduce the risk of inaccurate performance data, and provide a sufficient  level of confidence to Congress and the public that the information  presented is credible as appropriate to its intended use. As we have  previously reported, FEMA’s reliance on self-reported data from states  and lack of verification for FEMA preparedness grants has presented  reliability concerns. While AARs are also self-reported, they use a  narrative format that includes analyses of the various aspects of actions  following an exercise or actual event. State officials from two states we  spoke with said that they review AARs within their state in order to  evaluate NIMS implementation. For example, officials from one state said  they use AARs to identify gaps in NIMS implementation so they can align  resources to support better implementation. This is because exercises  enable stakeholders to identify both capability gaps and areas for  improvement. Further, the purpose of exercises is to provide a low-risk  environment to test capabilities, familiarize personnel with roles and  responsibilities, and foster meaningful interaction and communication  across organizations. As such, evaluating AARs could provide an  additional opportunity to look for areas of NIMS implementation that could  be improved.
According to NIC officials, they meet the requirements of HSPD-5 through  ongoing discussions with NIMS stakeholders, including state emergency  managers, to ensure ongoing management and maintenance of NIMS,  and by obtaining self-assessment information provided by states on NIMS  implementation in the URT that provides standards and guidelines.  Additionally, in order to meet the requirements of HSPD-5, in 2006 the  NIC developed implementation guidance through the NIMS Compliance  Objectives and Metrics for states and territories, tribal nations, and local  governments, which was last updated in 2010. However, NIC officials  said that, given the large number of states and their jurisdictions that have  adopted NIMS, any efforts to validate the information states provide on  their implementation efforts would be prohibitively difficult and costly.  While verifying each state and jurisdiction’s self-reported information may  not be feasible, states are already required to develop AARs under the  EPMG program, which could provide a source in addition to the URT by  which FEMA could assess progress made on implementation of NIMS, as  the 2013 assessment by the National Exercise Division demonstrates.  States are already required to develop AARs following their preparedness  exercises, based on grant program requirements. Moreover, FEMA’s  regional offices have multiple positions with responsibilities for NIMS  implementation, as well as preparedness exercises. Therefore,  developing policies and procedures for regional office staff to review and  assess information on states’ NIMS implementation, gleaned through a  review of exercise AARs, would not only help FPCs and NIMS  coordinators better meet their responsibilities for supporting and  enhancing regional preparedness, but also provide the NIC a means, in  addition to the URT, to further evaluate states’ self-reported assessments  of NIMS implementation.
section_title: Additional Opportunities Exist for FEMA Regional Offices to Enhance Collaboration with Regional Advisory Council Members
subsections: section_title: RAC Members and Liaisons View RAC Meetings as Useful and Well Run
paragraphs: As described earlier, the Post-Katrina Act required that each FEMA  region establish a Regional Advisory Council (RAC) in order to provide  the types of advice on emergency preparedness in their regions listed in  the Post-Katrina Act to the FEMA regional administrators. RAC  members who responded to our survey reported that they found RAC  meetings to be a useful collaborative experience, allowing them to obtain  good information, have interactions with other RAC members and FEMA  officials, and to offer advice to FEMA on regional emergency-related  issues. For example:
About 90 percent said that the RAC meetings were moderately or very  effective in providing information on how to make the region better  prepared for emergencies; the information communicated at RAC  meetings was moderately or very effective (89 percent); coordination  between the RAC and the regional office was moderately or very  effective (89 percent); the quality of the communications from their  FEMA regional office was good or excellent (91 percent); their  regional office was effective in asking members about their concerns  or suggestions about emergency preparedness issues in their region  (93 percent); and their participation was encouraged (93 percent).
About 81 percent said that the meetings provide a useful forum in  which to offer advice to FEMA on regional emergency-related issues;  most said that they likely would serve another term, if invited (81  percent); the overall preparedness of the region increased moderately  or greatly as a result of RAC meetings (69 percent); and, in general,  the meetings were well run, on time, on topic, and relevant to regional  emergency-related issues (89 percent).
Many of those FEMA regional offices that convene RAC meetings (see  below for a discussion of the variations in the occurrence of meetings)  prepare and send out agendas in advance of the meetings and  collaboratively involve RAC members in planning. Conducting early  outreach to participants and stakeholders to identify shared interests is an  effective collaborative implementation approach identified in GAO’s  Implementation Approaches Used to Enhance Collaboration in  Interagency Groups. RAC members who responded to our survey  reported that they appreciated the information provided to prepare them  for the RAC meetings. For example, 96 percent reported that they were  notified in advance about the meetings, 86 percent stated that the  agendas provided information necessary for preparing for the RAC  meetings, and 79 percent believed that distributing the minutes after the  meeting was a moderately or very effective approach.
section_title: Some FEMA Regions Do Not Routinely Meet with RACs to Collaborate on Emergency Preparedness
paragraphs: Although the RAC charters adopted by each of the 10 regions require that  their RACs meet twice annually, some RACs have not met routinely in  recent years to collaborate with FEMA regional officials to provide advice  and input from regional stakeholders. The DHS Inspector General  reviewed the FEMA Region IX Office in 2012 and the Region V Office in  2015 and concluded that because the RACs in those regions had not met  for a period of years, FEMA’s regional offices could be “missing  opportunities to identify and remediate weaknesses or deficiencies in  preparedness, protection, response, recovery, and mitigation activities.”  Figure 2 summarizes the number of RAC meetings in each region since  2007 through August 2015 (including both in-person meetings and  meetings conducted as interactive webinars).
One region cited the fact that they did not have funds to reimburse RAC  members for travel expenses, and another cited travel costs as a problem  for RAC members in a region as geographically large as theirs. Another  region stated that while they paid for the travel expenses, this necessarily  had to come from their available funds and that therefore, it meant having  less for other potential needs.
In our survey of RAC members, respondents cited several reasons why  they did not attend RAC meetings. For example, 68 percent of  respondents cited scheduling commitments and conflicts, 26 percent cited  insufficient or no funding for travel, and 11 percent said that the meeting  locations were inconvenient. Some RAC members said that their RAC  should definitely or probably meet more frequently (41 percent), while  others did not (30 percent said “probably” or “definitely not” and 23  percent said they were unsure or had no opinion). In open-ended  comments on RAC meetings attended by RAC members, 6 RAC  members stated that they would prefer having more in-person meetings  versus teleconferences or videoconferences.
FEMA Region V, in which the RAC did not meet for several years, took  steps starting in August 2014 to address a lack of interest from the  majority of the members. For example, regional officials invited RAC  members to other regional meetings with other federal agencies—a  process used by some other regions. In 2015, the regional office resumed  having RAC meetings, with one taking place in mid-January with 11 of 12  members attending. According to the regional RAC liaison, the high  attendance was the result of the regional administrator having personally  reached out to the individuals who had been recommended for RAC  membership and inviting them to participate. He said this individualized  communication with each person on the team helped to instill interest and  motivate involvement in the meeting, in the view of the liaison, and that  the personal involvement of the regional administrator makes the RAC  more effective.
The Post-Katrina Act requires FEMA regional offices to establish RACs in  order to provide input to each regional administrator on regional  emergency management issues by identifying any geographic,  demographic, or other characteristics peculiar to any state, local, or tribal  government within the region that might make preparedness, protection,  response, recovery, or mitigation more complicated or difficult, among  other things. Additionally, GAO’s Key Considerations for Implementing  Interagency Collaborative Mechanisms identified meetings that bring  stakeholders together as an effective collaborative practice, noting  particularly that such relationship-building is vital in responding to  disasters.
By routinely obtaining input from RAC members—whether through in- person meetings or via remote connection or other means—FEMA  regional offices could better ensure they are identifying geographic,  demographic, or other issues within the region to enhance emergency  preparedness. Additionally, for those FEMA regional offices that face  participation challenges, assessing the reasons for these barriers and  identifying targeted solutions, such as consistently offering remote  participation, or, as used by one region, direct communication from the  regional administrator to RAC members, could better ensure they are  able to obtain such input from the RACs effectively and on a regular  basis.
section_title: FEMA Regional Offices Are Responsive to RAC Suggestions but Do Not Regularly Provide Feedback on Their Recommendations
paragraphs: RAC members who responded to our survey generally indicated that  FEMA regions were responsive to their suggestions.
Seventy-five percent of the RAC members said that they had made  suggestions concerning issues related to emergency management at  RAC meetings or between RAC meetings; 79 percent stated that their  overall sense that FEMA was listening to their concerns increased  moderately or greatly during the time on which they had served on  their RAC; and 74 percent stated that FEMA was acting to address  their concerns.
Sixty-seven percent of the respondents said FEMA had taken action  to a moderate, great, or very great extent in response to suggestions  made by RAC members.
FEMA RAC liaisons we contacted by e-mail reported having taken action  in response to RAC member suggestions. Specifically, 9 of the 10 FEMA  RAC liaisons reported that input from their RACs had either led to  changes in specific FEMA policies or led to generally better informing  FEMA about members’ concerns. For example, at the suggestion of  their RACs, regional offices created a regional training and exercise  coordination website to share information; forwarded a suggestion from a  RAC to revise FEMA national policy to include ice storms as a  reimbursable emergency; and developed and distributed monthly  preparedness themes to help synchronize federal, state, and local  officials’ focus on common topics, among other things.
Although some FEMA regional offices have incorporated various  recommendations made by RAC members, 50 percent of the  respondents in our survey of RAC members said that FEMA should  provide more detailed feedback to them on suggestions or advice they  made at RAC meetings. Of the 10 regional offices, 7 reported that they  tracked outcomes of RAC efforts; 3 said they posted suggestions on  FEMA’s internal website (which is not accessible to non-FEMA  employees, including RAC members), and 4 said they used e-mail to  forward meeting notes to RAC members. One RAC member from our  survey commented on the need for a tracking system for the status and  resolution of issues raised, and another RAC member commented on the  need for better follow-up on recommendations. Developing a mechanism,  such as a Listserve or a dedicated website, to periodically notify all RAC  members in all 10 regions about the status of suggestions made by RACs  to regional administrators or to FEMA could help FEMA keep RAC  members aware of the status of their recommendations and thereby  address the concerns of those who wanted more detailed feedback.
In March 2014, FEMA’s NAC identified the need for FEMA to re-examine  the outcomes of its existing regional committees and councils, including  the RACs. Reporting the outcomes of RACs’ suggestions to the NAC  could help FEMA address this identified need. The NAC’s  recommendation to FEMA also reflects the key considerations for  implementing interagency collaborative mechanisms found in GAO’s  Implementation Approaches Used to Enhance Collaboration in  Interagency Groups. As the report stated, many of the meaningful  results that the federal government seeks to achieve, such as national  preparedness, require the coordinated efforts of more than one federal  agency, level of government, or sector. Implementation approaches for  interagency collaborative mechanisms include developing a plan to  communicate outcomes and track progress, and developing methods to  report on the group’s progress that are open and transparent. As noted  previously, while some regions have taken steps to track outcomes of  RAC efforts, none of the regional offices has a process for regularly  providing feedback on the status of recommendations. Providing  feedback to RAC members on their suggestions could enhance FEMA  regional offices’ collaboration with their RACs.
section_title: Although Many RAC Members Say Their Concerns Are Communicated to the NAC, Others Want More Feedback on Responses to Suggestions
paragraphs: The NAC, established by the Post-Katrina Act, incorporates state, local,  and tribal governments, nonprofit, and private sector input with regard to  improving emergency preparedness and response to natural and  manmade emergencies. Although almost two-thirds of RAC members  who responded to our survey said they believed their concerns were  communicated by FEMA to the NAC, some RAC members commented  that they wanted more feedback on the NAC’s responses to suggestions.  However, FEMA has not established policies and procedures to  communicate RACs’ regional concerns directly to their national-level  counterparts on the NAC, as proposed in 2011. Almost two-thirds of  respondents said FEMA was (very or moderately) effective in  communicating RAC concerns and issues to the NAC. However, in open  ended comments, six RAC members stated they that would like to receive  more information about the NAC and what it was addressing.
In September 2010, a FEMA local, state, tribal, and federal task force for  improving national emergency preparedness recommended that  communications between the RACs and the NAC be more robust.  Specifically, the task force recommended that RACs receive policy  information briefings directly from the NAC and provide input directly to  the NAC in order to provide local, state, tribal, and territorial officials with  influence across stages of the preparedness policy process and to create  an informational exchange between the NAC and the RACs to make  local, state, tribal, and territorial officials fuller partners in the  preparedness policy process.
In June 2011, FEMA responded to the recommendation from the task  force with an implementation plan, stating that, by November 24, 2011, it  would revise its policies and procedures to systematically link the RACs  to each other, the NAC, and FEMA headquarters, to help RACs influence  national emergency management policy. FEMA officials said they held a  meeting of all the RAC regional liaisons in August 2011 to propose  options to create a “NAC-RAC interface,” among other things. However,  they did not identify any additional subsequent actions they had taken to  implement the plan. For example, although FEMA stated that the August  2011 RAC liaison meeting had resulted in adding a “RAC Report Out” by  a regional administrator as a standing agenda item for the NAC meetings,  none of the NAC meeting agendas for any NAC meeting between August  2011 and September 2015 showed “RAC Report Out” as an agenda  item. In addition, our review of the minutes for all the NAC meetings  held between August 2011 and September 2015 found only an  occasional mention of issues of concern to RACs reported on by FEMA  regional administrators or others present.
Officials from FEMA’s Office of Regional Operations said that individual  RACs can transmit their concerns to the NAC, such as by letter or  through FEMA. The FEMA Office of the NAC provided a detailed  summary of the status of recommendations made by the NAC to FEMA  since 2008 to NAC members at the September 2015 NAC meeting.  However, this summary did not include the status of recommendations or  suggestions or communications made by RACs to the NAC. As such,  these processes do not appear to provide a systematic way in which to  ensure that issues raised in RAC meetings are consistently brought to the  attention of the NAC for its consideration or to provide a consistent  feedback mechanism for RAC members.
In November 2015, FEMA’s Office of the NAC said that although it is  precluded from establishing a formal mechanism between the RACs and  the NAC, it is working with FEMA’s Office of Regional Operations to  informally enhance coordination between the NAC and the RACs, such  as distributing NAC recommendation memoranda with the RACs for  informational purposes. Establishing policies and procedures, as  suggested by FEMA’s task force and 2011 implementation plan, to  enhance coordination and communication between the RACs and the  NAC could create greater opportunities for regional insights about  preparedness to inform national preparedness policies and a two-way  informational exchange between the NAC and the RACs to make local,  state, tribal, and territorial officials fuller partners in the preparedness  policy process.
section_title: Conclusions
paragraphs: Because of the key role FEMA’s regional offices play in national disaster  preparedness efforts, effective regional coordination and collaboration  between FEMA headquarters, FEMA regions, and state and local  stakeholders is essential. Multiple external and internal assessments and  our work have identified challenges, such as lack of coordination in grant- monitoring visits and inconsistent guidance, in FEMA’s grant  management. These challenges continue to hamper the effectiveness of  interactions between FEMA and state officials in implementing the  preparedness grants program. A plan with time frames, goals, metrics,  and milestones for addressing these challenges would better enable GPD  officials to monitor and improve their efforts to resolve long-standing  problems in coordination of monitoring activities and the consistency of  guidance given to state grantees. Regarding oversight of FEMA’s efforts  to implement the national system for managing incidents, there are  opportunities to access more sophisticated and comprehensive  information using FEMA regional staff to better assess NIMS  implementation. Further, using AARs that are already required to be  developed and submitted could provide a better opportunity to assess  NIMS implementation, including areas of success as well as areas where  improvement is needed. However, without policies and procedures that  call for systematically evaluating these reports, regional offices and the  NIC are missing a key opportunity to better assess national NIMS  implementation, which could enhance regional preparedness. Finally,  FEMA regional offices could more consistently and systematically take  steps to fully leverage RACs that can provide important inputs to national  preparedness efforts. For example, by ensuring that RACs provide  feedback to RAC members on the status of their recommendations,  FEMA could enhance RACs’ collaborative efforts and satisfaction of RAC  members. Similarly, by enhancing RACs’ connectivity with the NAC,  FEMA could increase the potential value of regional coordination and  collaboration and the resulting contributions to regional preparedness  efforts.
section_title: Recommendations for Executive Action
paragraphs: To promote more effective grant management coordination, the Secretary  of Homeland Security should direct the FEMA Administrator to develop a  plan with time frames, goals, metrics and milestones detailing how GPD  intends to resolve longstanding challenges associated with its existing  hybrid grants management model, which divides responsibilities between  regional and headquarters staff.
To enable more sophisticated and comprehensive awareness of states’  NIMS implementation, the Secretary of Homeland Security should direct  the FEMA Administrator to develop policies and procedures for regional  staff to review AARs from preparedness exercises within their region, and  headquarters staff to review these evaluations in order to have a better  understanding of NIMS implementation.
To enhance the value of RACs, the Secretary of Homeland Security  should direct the FEMA Administrator to take the following three actions:  1.  Ensure that all regional offices routinely obtain input—whether in  person, by teleconference or by other remote connection—from their  RAC members on ways to enhance overall emergency preparedness  in their regions. In cases where RAC member participation is low,  regional offices should assess and identify targeted solutions for  increasing member participation, such as offering remote participation  or alternative forums;  2.  Develop a mechanism to update RAC members on the status of  recommendations made by RACs to FEMA; and  3.  Establish processes for enhanced coordination and communication  between the RACs and the NAC.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for comment. DHS provided  technical comments, which we incorporated as appropriate. On January  15, 2016, DHS also provided written comments, reproduced in full in  appendix IV. DHS concurred with four of our recommendations and did  not concur with one.
DHS did not concur with our first recommendation that the Secretary of  Homeland Security direct the FEMA administrator to develop a plan with  time frames, goals, metrics, and milestones detailing how GPD intends to  resolve longstanding challenges associated with its existing hybrid grants  management model. In its comments, DHS stated that it disagreed with  GAO’s characterization of longstanding challenges in managing  preparedness grants. As we stated in the report, multiple assessments  dating back to 2009 have reported challenges with the hybrid model that  splits management of preparedness grants between FEMA’s  headquarters and regional offices. As also noted in the report, officials  from 4 FEMA regional offices and officials from 3 states within those  regions provided numerous examples of a lack of coordination between  headquarters and regional staff in managing preparedness grants,  including instances that took place in 2014 and as recently as September  2015. Based on our review of the past assessments and the audit work  we performed, we believe that these long-standing challenges have yet to  be resolved.
In its written comments, DHS also stated that FEMA already empowers  its ten regions to coordinate and communicate with headquarters and  other regional offices to facilitate working relationships. While FEMA has  taken various actions since 2008 to improve coordination related to  preparedness grants—some actions as recent as February 2015—these  steps have not resolved these challenges, as described in the report.
When we spoke with GPD officials in September 2015, they agreed that  there were continuing opportunities to improve coordination and  communications in their management of preparedness grants. Therefore,  we continue to believe that FEMA would benefit from a more strategic  approach that a plan, with time frames, goals, metrics, and milestones  detailing how officials intend to resolve longstanding challenges  associated with the existing hybrid model, could provide.
DHS also expressed concern about the wording of our recommendation  and how the recommendation, as worded, could be closed. Specifically,  DHS stated that FEMA is not aware of nor did the recommendation  provide any specific criteria that could be used to gauge the success of  any proposed plan. While we recommended that DHS develop a plan with  time frames, goals, metrics, and milestones for resolving its longstanding  coordination challenges, we did not prescribe specific actions to address  these challenges. However, our work across the federal government has  identified implementation approaches used to enhance collaboration that  could inform GPD officials’ efforts to clearly define short-term and long- term outcomes and track and monitor their progress. Nevertheless,  FEMA could implement this recommendation by developing a plan and  we believe that doing so would address the intent of this  recommendation.
Regarding our second recommendation that the Secretary of Homeland  Security direct the FEMA Administrator to develop policies and  procedures for regional staff to review AARs from preparedness  exercises within their region, and headquarters staff to review these  evaluations in order to have a better understanding of NIMS  implementation, DHS concurred and said it recognizes that joint efforts by  regional and headquarters staff are essential to enable more  sophisticated and comprehensive awareness of NIMS implementation.  However, DHS expressed concerns about overstating the value of AARs  as a means to improve understanding of NIMS implementation.  Specifically, DHS stated that FEMA does not agree that the “best way to  assess how well NIMS has been implemented is by assessing states’  performance in preparedness exercises and real-world events.” As noted  in the report, most all of the officials we spoke with believed that the  current method of determining NIMS implementation—states self- certification via the URT—was not an effective method of determining  NIMS implementation, and officials in all of the FEMA regions and 8 of the  states we interviewed said that looking at how effectively a jurisdiction  responds during an exercise or real world event is the best way to assess  NIMS implementation. DHS also stated that the extent to which a  jurisdiction has implemented NIMS is nuanced and therefore impossible  to assess accurately from an AAR. The report does not suggest that  reviewing AARs would give a comprehensive and definitive assessment  of NIMS implementation but rather that it could provide FEMA an  additional way to assess and elaborate on the information states currently  provide. In fact, FEMA used AARs to conduct an assessment of NIMS in  2013, which resulted in some recommendations to help improve NIMS  implementation. Finally, as we have previously reported, the Post-Katrina  Act requires FEMA to carry out the National Exercise Program. The Act  established the Program, among other things, as a means for FEMA to  test and evaluate NIMS implementation efforts. As a result, we continue  to believe that AARs, an essential part of the National Exercise Program,  are an important part of FEMA’s efforts to assess NIMS implementation  and we added a reference to this requirement to our final report. DHS did  not identify any specific steps to address this recommendation but said  that the National Preparedness Directorate would work with regional  NIMS coordinators and the states to explore options during calendar year  2016, with potential changes to be implemented in calendar year 2017.
In its written comments, DHS concurred with our third, fourth, and fifth  recommendations related to enhancing the value of RACs. Specifically,  regarding the third recommendation that the Secretary of Homeland  Security direct the FEMA administrator to ensure that regional offices  routinely obtain input from their RAC members, DHS concurred and  stated that the regions will conduct two meetings a year with their RAC  members; that a RAC coordinator within each region will be responsible  for the oversight of these meetings; and that open dialogue will be  encouraged between RAC members and the regions, with each region  establishing a RAC coordinator to facilitate this communication. If  implemented as planned, these actions should help address the intent of  the recommendation to ensure that regional offices routinely obtain input  from their RAC.
Regarding the fourth recommendation that the Secretary of Homeland  Security direct the FEMA administrator to develop a mechanism to update  RAC members on the status of recommendations made by RACs to  FEMA, DHS concurred and stated that it will build into each RAC meeting  a briefing by regional personnel on the status of recommendations, if any,  made during the previous meeting. The regional administrator or RAC  coordinator within the region will be responsible for the brief. Additionally,  if RAC members are not satisfied with the implementation of RAC  recommendations, it will be expected that the regional office will provide  information on why a specific recommendation was not adopted or why  implementation is delayed. DHS further stated that each region may have  its own method of dispensing updates to its RAC members between  meetings, including monthly email updates, posting on their website, or  informational calls. If implemented as planned, these actions should help  address the intent of the recommendation to update RAC members on  the status of its recommendations to FEMA.
Regarding our fifth recommendation that the Secretary of Homeland  Security direct the FEMA administrator to establish processes for  enhanced coordination and communication between the RACs and the  NAC, DHS concurred and stated that regional administrators will continue  to attend NAC meetings and provide FEMA regional updates that include  RAC activities with the regions. Moreover, regional coordinators will  ensure that RAC members are aware of any NAC meeting occurring in  their own region, and that the RAC sharepoint site will be used as a  platform to update RAC coordinators on impending NAC meetings to  share with their RAC members. If implemented as planned, these actions  should help address the intent of the recommendation to establish  processes for enhanced coordination and communication between the  RACs and the NAC.
We are sending copies of this report to Secretary of Homeland Security,  appropriate congressional committees, and other interested parties. This  report is also available at no charge on GAO’s website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (404) 679-1875 or at CurrieC@gao.gov. Contact points for our  Office of Congressional Relations and Office of Public Affairs can be  found on the last page of this report. Other major contributors to this  report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this study were to assess the extent to which the  Federal Emergency Management Agency (FEMA) headquarters and  regional offices have (1) addressed grant management coordination  challenges between headquarters and the regions; (2) established a  comprehensive system to assess National Incident Management System  (NIMS) implementation; and (3) collaborated with Regional Advisory  Council (RAC) members.
To address the first objective, we gathered and reviewed relevant  documentation, such as FEMA assessments of grants managed by Grant  Programs Directorate (GPD), FEMA’s previous delegations of grant  management responsibilities to regional offices, agency task force reports  on advantages and disadvantages of regionalization, FEMA headquarters  and regional office grant management roles and responsibilities, FEMA  documentation on regionalization pilot programs, and other memoranda  and internal documents. We also reviewed prior GAO reports on regional  preparedness, grant management, and consolidation of management  functions. We interviewed FEMA’s Director of the Preparedness Grants  Division within GPD and other senior GPD officials to discuss  programmatic grant management as well as previous FEMA assessments  of the impact of moving management responsibilities to FEMA regional  offices. We also interviewed grant management officials from 4 FEMA  regional offices—Regions III, IV, VI, and IX—to discuss financial grant  management and coordination of monitoring activities with GPD. These  offices were selected in order to provide a mix of geographic locations,  types of disasters typically experienced, and whether the regional office  had been selected to participate in FEMA grant management pilot  programs. Similarly, we interviewed officials from 9 states or territories  (Arkansas, California, Florida, Georgia, Louisiana, Nevada, Texas,  Virginia, and Washington, D.C.) from the 4 FEMA regions to discuss grant  management within the states as well as interactions with FEMA grant  management officials. These states were selected to reflect a diversity of  experiences within the 4 FEMA regions we chose. While the information  gained from these interviews cannot be generalized across all states, it  provides useful insights into the nature of FEMA GPD and regional  coordination with states with regard to grant management.
To address the second objective, we reviewed NIMS-related documents  such as the NIMS Doctrine, NIMS Compliance Guidance, NIMS  implementation reports, and results of the NIMS implementation  questions in the Unified Reporting Tool (URT). We also reviewed  documents on the Homeland Security Exercise and Evaluation Program  (HSEEP) and on preparedness grants requirements. We compared  FEMA’s efforts in NIMS implementation and verification with Homeland  Security Presidential Directive-5 and Office of Management and Budget  (OMB) Circular No. A-11, regarding data validation and verification. We  interviewed officials at FEMA’s National Preparedness Directorate, which  includes the National Integration Center (NIC)—the office responsible for  NIMS implementation—to discuss the NIC’s efforts in implementing  NIMS. To discuss NIMS implementation efforts at the regional and state  levels, interactions between regional and state officials, and mecha nisms  for verifying how well NIMS is being implemented, we also interviewed  officials from the 4 FEMA regional offices and the 9 states or territories  mentioned above. In addition, because of the recommendations of  officials in several regional offices and states, we interviewed an official  from a 10th state (Oklahoma) from within the selected FEMA regions.  Again, while the information gained from these interviews cannot be  generalized across all states, it provided useful insights into the various  efforts in implementing NIMS. Additionally, we analyzed all 35 full-scale  exercise after-action reports (AAR) that FEMA received for exercises  conducted during fiscal year 2014 and in the 4 regions we visited. We  used the AARs as they provide qualitative analysis of outcomes of  exercises, which are conducted in order to identify capability gaps and  areas for improvement. The purpose of the analysis was to determine if  they could be used to assess the level of participants’ implementation of  NIMS. We did this by developing a crosswalk that linked the six NIMS  elements listed in the URT, and that states should incorporate into  exercises, to a core capability—the unit by which exercises are evaluated.  In developing this crosswalk, we found three core capabilities that we  used as proxies for the six NIMS elements that are to be incorporated into  exercises—operational coordination, operational communication, and  public information and warning. While aspects of NIMS implementation  could be gleaned from other core capabilities, for the purposes of this  analysis we used the three core capabilities identified through our  crosswalk. Of the 35 reports we analyzed, 33 assessed at least one, if not  two or all three, of the core capabilities we identified as proxies. In those  reports where one or more of the core capabilities were addressed, we  looked to see if there were issues associated with those core capabilities,  and the nature and extent of the issues, by reviewing various sections of  the reports. For the core capability of operational coordination, we also  looked at the exercise objectives to see if they aligned with some element  of NIMS components. If an AAR reported that the core capability that we  assessed either was performed with some challenges, was performed  with major challenges, or was unable to be performed, we included it in  our count of a capability that encountered some issues. From this we  could glean the extent to which various NIMS elements are working well  and those that still have room for improvement. While results of these  exercises are not generalizable to the country as a whole, they do provide  useful insights into NIMS implementation in some areas.
To address the third objective, we interviewed FEMA headquarters and  regional office officials and analyzed FEMA and Department of Homeland  Security (DHS) Office of Inspector General (OIG) reports with information  and findings relevant to RACs. We attended two RAC meetings in 2  regions that took place during the early part of our review, and obtained  documentation of agendas and other relevant materials for RAC meetings  that occurred between 2007 and 2014 for the RACs in all 10 FEMA  regions. We also reviewed agendas and meeting minutes for NAC  meetings that took place between August 2011 and September 2015. We  reviewed relevant federal laws and regulations, federal internal controls  standards, and leading practices for program management.
To identify the extent to which RAC members view their RAC experiences  as effective in conveying their concerns to FEMA, we conducted a self- administered, web-based questionnaire survey of RAC members of each  FEMA region’s RAC; FEMA’s 10 regional offices identified an initial  population of 133 active RAC members. (See app. II for a copy of the  survey questionnaire and the results.)
Using e-mail addresses provided by each of the FEMA regions for their  RAC members (which included the RAC members’ names and, in most  cases, their place of work or other occupational identifier), we e-mailed  each RAC member a link to a secure survey website, along with a unique  identifier and password to control access to each member’s  questionnaire. Most survey questions were closed-ended, in which RAC  members selected from a list of possible responses. To obtain additional  narrative and supporting context, survey respondents were given  opportunities to provide additional open-ended comments throughout the  survey.
The survey began on May 27, 2015, and data collection ended on June  29, 2015. We e-mailed the survey link to 126 of the RAC members  identified by FEMA regional offices. Of these 126 e-mail contacts, e-mails  to 14 were initially undeliverable and 2 were e-mails for FEMA employees  who had been incorrectly listed as RAC members (FEMA employees  cannot be RAC members). Of the 14 non-FEMA RAC members, we  determined that 12 were no longer RAC members. Of the remaining 2,  one informed us that she had only recently been appointed to a RAC and  had not attended any RAC meetings, and the other told us in a follow-up  telephone call that he had not been a RAC member for at least 4 years.  On the basis of this information about RAC membership status, we  defined all 16 cases as ineligible and excluded them from the survey  population. Of the 110 contacted that were still presumed to be eligible  members, 77 responded to the survey, resulting in a 70 percent response  rate. Of the 77 respondents, 3 answered an initial screening question  stating that they were not currently members of the RAC in their region,  which left a total of 74 respondents whose responses were usable for our  analysis of the rest of the questions. During the course of survey data  collection, we sent periodic reminder e-mails to all nonrespondents to  encourage participation in the survey. We also conducted tailored follow- ups by e-mail or telephone with selected nonrespondents, for example,  those only partially completing their questionnaires.
Because our survey was designed to include the entire population of RAC  members, our results are not subject to sampling error. However, the  practical difficulties of conducting any questionnaire survey may introduce  a variety of non-sampling errors. For example, differences in how a  particular question is interpreted, the sources of information available to  respondents, or the types of people who do not respond to the survey or  a particular question can introduce error into survey results. We included  steps in the survey design, data collection, and analysis stages to  minimize these types of non-sampling errors. To minimize measurement  error and nonresponse error, we designed draft questionnaires in  collaboration with GAO survey specialists. We conducted pretests by  telephone with three RAC members from three regions. On the basis of  survey specialist input and these pretests, we made revisions to  questionnaire drafts as necessary to reduce the likelihood of  measurement and nonresponse errors (the types of nonresponse  associated with the perceived burden, lack of question clarity or relevance  to the respondent). In addition, our analysts answered respondent  questions and resolved difficulties that respondents had in completing our  questionnaire. We asked open-ended comment questions at various  points in the questionnaire to allow respondents to explain or provide  context for their answers, which helped inform and corroborate our  interpretation and analysis of the survey results. We examined survey  results to identify and remediate any inconsistencies or other indications  of response error. Finally, we made multiple follow-up contacts with non- respondents throughout data collection to reduce nonresponse. To  minimize the possibility of data processing error, a second data analyst  independently verified the accuracy of all computer analyses. In addition,  RAC members made their responses directly into an automated web  survey instrument, preventing errors associated with manual data entry of  written answers.
We conducted our work from October 2014 to February 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Results of GAO Survey of Federal Emergency Management Agency (FEMA) Regional Advisory Council Members
paragraphs: Below is the text of the survey as it was sent to Regional Advisory Council  (RAC) members. Following the cover page, the questions are shown,  along with the results in percentages. Narrative answers to open-ended  text questions are not displayed to prevent the identification of individual  respondents. Percentages may not always sum to 100 percent because  of rounding of decimal figures.
section_title: Appendix III: National Incident Management System (NIMS) Implementation Questions Asked of States and Territories in the Unified Reporting Tool
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Chris Keisling (Assistant  Director), David Alexander, Chuck Bausell, David Dornisch, Eric  Hauswirth, Susan Hsu, Valerie Kasindi, Stuart Kaufman, Tracey King,  Marvin McGill, Carl Ramirez, Robert Rivas, and Jonathan Tumin made  key contributions to this report.