Federal Emergency Management Agency: Strengthening Regional Coordination Could Enhance Preparedness Efforts
Released: 2016-02-04T00:00:00

Letter

Background

Preparedness Grant Management

FEMA preparedness grant management is composed of different financial and programmatic management functions. FEMA created GPD in April 2007, to consolidate oversight of all FEMA grants. GPD, in FEMA headquarters, provides subject matter expertise in response to regional office and stakeholder inquiries; develops grant guidance; and delivers policy, training, and systems and data analysis. The 10 FEMA regions are responsible for financial monitoring, implementing corrective action plans, as well as other follow-up activities with grantees. Financial management functions include monitoring grantee expenditures, amending financial terms of grants, and closing out grants. Program management includes developing grant application packages creating program standards, and monitoring grantee activities to ensure alignment with homeland security strategies.

FEMA’s regional offices have program management responsibility for various legacy programs primarily because they managed those programs prior to the establishment of GPD. The largest of these legacy programs is the Emergency Management Performance Grant (EMPG), which provided over $350 million in fiscal year 2015 to assist local, tribal, territorial, and state governments in enhancing and sustaining all-hazards emergency management capabilities. GPD has management responsibility for the Homeland Security Grant Program, which provides more than $1 billion annually to help strengthen the nation against risks associated with acts of terrorism and other catastrophic events.

National Incident Management System

FEMA works with state and local stakeholders to implement NIMS, a comprehensive, national approach to incident management that is applicable at all jurisdictional levels and across functional disciplines and the full spectrum of potential incidents, hazards, and impacts, regardless of size, location, or complexity. DHS established NIMS in 2004 to improve coordination and cooperation between public and private entities in a variety of incident management activities and provide a common standard for overall incident management. NIMS implementation consists of training, using a standardized way of defining and categorizing emergency resources known as resource typing, and incorporating a standardized organizational emergency response structure called the Incident Command System into emergency management plans, policies and procedures, among other things. States, tribes, and local jurisdictions are required to adopt and self-assess their level of NIMS implementation, and certify their compliance and report it to FEMA through the URT in order to receive preparedness grant funding from FEMA. Additionally, states receiving preparedness grants are required to develop and maintain an exercise program consistent to the degree practical with HSEEP in support of the National Exercise Program. HSEEP exercises identify exercise objectives and align them to FEMA’s 32 “core capabilities” for evaluation during the exercise. In order to report on the required exercises, states receiving preparedness grants are required to prepare after-action reports—evaluations of performance—on core capabilities assessed in the exercise, and to submit those to FEMA.

FEMA established the NIC in 2004. The NIC, a division of FEMA’s National Preparedness Directorate, is responsible for the ongoing management and maintenance of NIMS, including developing guidance to define and categorize the resources and job position qualifications requested, deployed, and used in incidents. Specifically, the NIC is to coordinate with state and local stakeholders to develop national standards, guidelines, and protocols for incident management.

In 2008, FEMA established new regional positions for federal preparedness coordinators (FPC), who direct and coordinate the activities of each regional national preparedness division to ensure regional implementation of national preparedness programs, policies, goals, and objectives. FPCs are to evaluate preparedness program activities to assess their effectiveness; changes in risk or capability; and the performance of preparedness program activities including after-action reports, training evaluations, lessons learned, and corrective actions. For example, FPCs are responsible for monitoring NIMS compliance and implementation, in addition to other national preparedness initiatives. Further, FPCs are to proactively engage stakeholders and acquire an understanding of their preparedness efforts and also assist in the planning, design, execution, and evaluation of federal, state, local, and regional exercises. FEMA regional NIMS coordinators act as subject matter experts regarding NIMS for the local, state, territorial, and tribal nation governments within their FEMA regions, as well as for the FEMA regional administrator and staff.

Regional Advisory Councils

The Post-Katrina Act requires that each FEMA regional administrator establish a RAC with members from state, local, and tribal entities to provide advice on emergency management issues specific to the region. While the Post-Katrina Act does not specify how many members a RAC should have, the law states that a state, local, or tribal government located within the geographic area served by the regional office may nominate officials, including emergency managers, to serve as members. Each of the 10 FEMA regional offices established a RAC after the Post- Katrina Act was enacted in 2006. In August 2011, FEMA developed a RAC charter template that each of the regional offices voluntarily adopted; the charters require that RACs meet twice annually in order to provide the types of advice on emergency preparedness in their regions to the FEMA regional administrators listed in the Post-Katrina Act.

In addition to the RACs, the Post-Katrina Act also required that FEMA establish the National Advisory Council (NAC) to ensure effective and ongoing coordination of federal preparedness efforts, among other things. The NAC advises the FEMA Administrator on all aspects of emergency management, and incorporates state, local, and tribal governments; nonprofit; and private sector input in the development and revision of the National Preparedness Goal, the National Preparedness System, the National Incident Management System, and other related plans and strategies.

FEMA’s 10 regional offices are the principal conduit for delivery of preparedness programs and activities to state, tribal, and local partners; non-governmental organizations; the private sector; and citizens. Such preparedness activities include, among other things, stakeholder coordination and information sharing, consulting, planning support, capability assessments and reporting, exercise performance and evaluation, and internal and external training. Figure 1 shows FEMA’s 10 regions and the states and territories that compose each region (the District of Columbia is in Region III).

GPD Lacks a Plan to Address Long- standing Challenges in Coordinating Preparedness Grants between FEMA Headquarters and Regions

Past Assessments Supported Moving Grant Management Functions to the Regions to Address Oversight and Coordination Problems, but FEMA Decided Not to Change Its Approach

Currently FEMA uses a hybrid management model for preparedness grant programs, with shared responsibilities between headquarters and the regions. In October 2009, in response to a congressional request, the National Academy of Public Administration (NAPA) conducted a study and recommended that FEMA transfer all preparedness grant authorities to its regional offices. Similarly, in May 2010, FEMA’s Office of Policy and Program Analysis (OPPA) reported that FEMA’s hybrid grants management model created significant inconsistencies in managing preparedness grants, and recommended that programmatic management functions for preparedness grants should be transferred to the regional offices. In September 2010, GPD’s Regional Implementation of Grants (RIG) task force also reported that programmatic grant management functions for grants should be moved to the 10 regional offices, because of considerations of customer service, efficiency, people, processes, and systems. In July 2011, we reported that GPD’s efforts to regionalize management responsibilities for preparedness grants were consistent with internal control standards. On the basis of the results of our review of FEMA’s plans and efforts to regionalize grant management functions, we did not make recommendations at that time.

FEMA’s Deputy Administrator for Protection and National Preparedness, in consultation with FEMA’s Administrator, decided in early 2012 against pursuing further regionalization of grant management functions, according to FEMA’s Assistant Administrator for Grant Programs. The Assistant Administrator said their decision was based on, among other things, estimates that the costs of regionalization would be greater than the annual savings identified in the earlier OPPA study and FEMA management’s belief that risks associated with the change, such as inconsistent program implementation across the regions, outweighed the potential benefits. As a result, FEMA continues to use its hybrid management model for preparedness grant programs, with shared responsibilities between headquarters and the regions.

FEMA Regional Offices and Grantees Identified Challenges with FEMA’s Hybrid Management Model

Long-standing challenges associated with the separation of management functions under the hybrid model of grant management, including the lack of coordination of grant-monitoring visits, and coordination and implementation of guidance provided by headquarters officials to state officials, continue to create challenges.

In 2010, GPD’s RIG task force report cited examples of how programmatic and financial monitoring visits, conducted by FEMA GPD staff and regional offices respectively, occurred independently of one another and were disruptive to the state emergency management agency’s day-to-day operations. In response to the lack of coordination in monitoring site visits, GPD’s Deputy Assistant Administrator sent a memorandum to staff in 2011 citing concerns that it had become “apparent GPD HQ staff continue to do program monitoring visits without proper notification to the regional offices” and directing that headquarters personnel conduct all monitoring visits in conjunction with regional staff, changing the dates of any planned headquarters visits to accommodate regional staff schedules. While the memorandum called for disciplinary action against any GPD personnel who did not follow the directive, GPD officials did not identify any instances where disciplinary action had been taken.

We also found challenges in the coordination of site visits. Officials in 3 of 9 states we talked with on grants issues stated that monitoring visits from FEMA GPD and the regional offices do not appear to be well coordinated. For example, officials in 1 state said that in September 2014, GPD and regional grants staff conducted site visits during the same week; however, neither the GPD staff nor the regional staff knew that the other was conducting a monitoring visit that week. The state officials stated that they had to leave one monitoring meeting early to go to another monitoring meeting by a different group of grant management staff. Similarly, regional grants officials in 2 of the 4 regions we visited said in 2015 that they may or may not receive information on monitoring activities undertaken by GPD officials, depending on who is conducting the monitoring activities. A grant official in 1 region we visited stated that GPD conducted a site monitoring visit as recently as September 2015, but did not inform regional staff of this visit. Regional grants management officials in all 4 regions we visited stated that not having the results of programmatic monitoring creates challenges in financial monitoring. For example, financial monitoring may identify a significant drawdown of grant funds; however, without program information, regions cannot determine if this corresponds to significant progress in program implementation.

Further, challenges related to coordination and implementation of guidance provided by GPD (headquarters) officials, FEMA regional officials, and state officials has been a long-standing problem. For example, in 2009, NAPA identified regional challenges associated with assessment activities including grant monitoring, noting that FEMA regional and headquarters units “need to reduce burden on states and ensure assessments-requirements are coordinated among FEMA divisions and the federal sector” and that new documents from FEMA headquarters “typically have a short review and comment period, and adherence is often required prior to finalization.” Similarly, the May 2010 RIG Task Force Report noted that “the timeliness of communication and information distribution with the grantee community needs to be improved” and the need to “formally establish regional liaisons to support the consistent prioritization of communication within GPD and with grantees.”

We found that these issues related to coordinating guidance provided by headquarters officials to state officials continue to create challenges. Officials from all 4 FEMA regions we visited said that GPD officials had provided guidance to state grantees that contradicted guidance given by the regional staff. Further, officials in 2 of the 4 regional offices we visited stated that incorrect guidance given by GPD officials to state grantees resulted in deficiencies during financial monitoring conducted by regional grants personnel. Specifically, according to FEMA grant management officials in 1 region, a grantee, at the advice of GPD officials, shifted funds from one grant program into other grant programs, but regional grants personnel later found, as part of their financial monitoring, that this was a violation of regulations and required that the funding be returned. Similarly, FEMA grant officials in another region stated that a new grantee that received transit grants drew down the entire grant amount soon after the grant was awarded, which is a violation of FEMA grant management regulations. Regional officials asked the grantee to return the funds; however, the grantee stated that GPD officials had told him he could draw down these funds. Regional officials added that had the grantee not returned the funds, the next step would have been to do an official collection; however, regional offices do not have the authority to do this because the grant program is managed from FEMA headquarters by GPD. Officials from 3 of the 9 states we visited told us that they had received contradictory information from regional staff and GPD staff regarding the management of grant programs.

GPD officials agreed that there were continuing opportunities to improve coordination and communications in their management of preparedness grants and said that FEMA had taken steps during the course of our review to facilitate coordination and communication. For example, GPD officials said that they had taken steps to improve coordination of site visits and that, beginning in fiscal year 2013, GPD made integrated financial and programmatic monitoring a priority, and incorporated this into its monitoring plans. According to FEMA’s fiscal year 2014 grants monitoring plan, GPD officials are to contact regional grant staff when monitoring activities are initiated and document this interaction. GPD officials did not provide any documentation that such coordination had occurred. Nonetheless, they identified two joint GPD and regional office monitoring visits that took place during the summer of 2015 (at the request of the regions).

GPD officials also noted that, in a February 2015 meeting with regional officials, GPD and the regional offices agreed to the following: elevate grant-related issues to regional administrators and to the Assistant Administrator for Grant Programs, provide regional grants staff visibility on programmatic grant reviews performed by GPD staff, conduct desk reviews in coordination with the regions and provide copies of the final reports to the regional offices, provide GPD’s annual monitoring plan to the regional offices and conduct joint monitoring of the same states for programmatic and financial issues when possible, provide quarterly analytical reports of grant-monitoring-related issues to all regional offices, and schedule regular calls between GPD management and regional grants division directors.

These most recent efforts to improve coordination of preparedness grant management functions are not significantly different from the historical efforts GPD has taken. For example, GPD cited monthly coordination calls between GPD and the regional offices as a coordinating effort meant to address challenges we identified. However, FEMA also cited these calls as a coordination mechanism in 2012 and noted that these calls had been a recurring effort since 2008. Similarly, providing copies of GPD’s annual monitoring plan and copies of reviews performed by GPD do not appear to have resolved the underlying challenges associated with GPD’s hybrid grants management model.

GPD officials also said they intend to look at FEMA’s overall grants monitoring efforts to see what can be done to improve the program for fiscal years 2016 and beyond, including risk assessments, general assessments, on-site desk reviews, reports with real-time data and metrics, and corrective action plans. However, FEMA has not developed a plan with time frames, goals, metrics, or milestones for how and when it will address long-standing coordination challenges associated with the existing hybrid grants management model identified in previous FEMA assessments and in our review. According to FEMA’s Assistant Administrator, GPD has involved the regions in performance measure development since September 2011 and has communicated the measures and results to the regions. These performance measures include, among other things, percentage of preparedness grant funds monitored and percentage of grants award determined to be low risk. However, these performance measures do not address the deficiencies in coordination identified in our review.

According to effective program management practices, specific goals and objectives should be conceptualized, defined, and documented in the planning process, along with the appropriate steps, time frames, and milestones needed to achieve those results. Given the longstanding coordination challenges, establishing a plan with time frames, goals, metrics, and milestones could help FEMA headquarters and regional offices address these challenges.

FEMA Could Have Better Visibility into NIMS Implementation by Using the Results of Preparedness Exercises

The NIC Relies on States’ Self-assessments to Determine NIMS Implementation

The NIC uses a self-assessment tool to determine if all states and territories have adopted and implemented NIMS elements into their emergency response planning and training. The NIC relies on a self- assessment tool, as the Post-Katrina Act requires that FEMA develop an assessment system to determine compliance with NIMS but does not require that FEMA physically verify compliance. Since 2005, all 56 states and territories have been required to meet NIMS compliance and implementation requirements to be eligible to receive federal preparedness grants. The NIC began using the latest version of its tool for states to submit self-assessments—the URT—in 2013. In the URT spreadsheet, states and territories fill in responses to a series of questions aimed at determining the level of NIMS implementation. Generally, states and territories gather this information from their subjurisdictions and consolidate the information at a state level to report to FEMA. (For a list of questions, see app. III.)

Although states generally report high levels of NIMS implementation in their URTs, officials from all four FEMA regional offices we spoke with and from 9 of the 10 states we spoke with said that the NIMS self- assessments are perfunctory and do not necessarily measure whether, or how well, NIMS is being implemented. For example, in fiscal year 2013, 100 percent of the states and territories reported having “formally adopted” NIMS, and incorporated NIMS principles and concepts into training and exercises. Similarly, in fiscal year 2014, states and territories reported that 93 percent of their subjurisdictions had formally adopted NIMS, about 91 percent had incorporated NIMS concepts and principles into training, and about 93 percent incorporated NIMS into exercises. FEMA regional and state officials we spoke with said there is an incentive to report compliance with NIMS regardless of the actual status of implementation since it is a requirement for receiving preparedness grants. According to preparedness grant guidance, there is no requirement for states to verify NIMS compliance or implementation at the subjurisdiction level. However, of the states we spoke with on NIMS implementation, 1 had implemented its own NIMS compliance check program. Specifically, a state emergency management official in 1 state we spoke with said that his state has initiated its own reviews of NIMS implementation and found that some reporting of full NIMS implementation by subjurisdictions were questionable. For example, in one subjurisdiction, state officials found the following:

The subjurisdiction could not produce training records to show that emergency management personnel had received NIMS training.

The requirement to incorporate NIMS principles into the subjurisdiction’s emergency operations plans had not been completed.

There were no records of exercises or corrective action plans available to identify if NIMS had been incorporated into the subjurisdiction’s exercise program.

While this particular example cannot be generalized to other states, it shows that in some cases, subjurisdictions within one state with a NIMS compliance check program are reporting levels of NIMS implementation that may not fully reflect the status of local NIMS efforts.

Existing Assessments after Exercises and Real- World Events Provide a Means of Further Assessing States’ Self- reporting on NIMS Implementation

Officials from all 4 regional offices and from 8 of the 10 states we spoke with said that the best way to assess how well NIMS has been implemented is by assessing states’ performance in preparedness exercises and real-world events. They all agreed that if NIMS principles have been well implemented, responses to disasters will be more effective because the objective of NIMS is to ensure that all levels of government across the nation work together efficiently and effectively using a single, national approach to an incident. Further, according to the NIMS doctrine, using a comprehensive national approach improves the effectiveness of emergency management personnel’s responses to the full spectrum of potential incidents and hazard scenarios, including natural hazards, terrorist activities, and other manmade disasters. To capture information on states’ performance during preparedness exercises and real-world events, states generally develop after-action reports. The EMPG grant program, under which all states and territories receive funding, specifically requires participants to develop and maintain an exercise program, consistent to the degree practical with HSEEP in support of the National Exercise Program, and to conduct preparedness exercises and submit AARs that evaluate the state or territory’s performance during the exercises based on one or more core capabilities. Under HSEEP—which provides a common approach to designing, developing, conducting, and evaluating exercises—planners identify exercise objectives and align them to core capabilities. In some cases, the exercise objectives test one or more of the NIMS elements identified by FEMA that should be incorporated into training. As such, exercises could also allow for identifying areas of needed improvement in NIMS implementation, one of the legislative bases for establishing the National Exercise Program.

FEMA regional officials we spoke with stated that some of FEMA’s national core capabilities are good proxies for how well NIMS is being implemented, and that AARs—which states submit to the National Exercise Division—can provide insight into the level of NIMS implementation when certain of these core capabilities are assessed. NIC officials stated if an exercise objective includes testing an element of NIMS, then it is possible to assess NIMS implementation. For example, evaluating a state’s performance related to the core capability of operational coordination could include as an exercise objective to evaluate the command and control of an incident according to NIMS principles. Similarly, the core capability for operational communications could provide an assessment related to the NIMS elements for interoperability of compatible communications, technology, and information management.

We analyzed AARs for 35 full-scale exercises that were conducted in fiscal year 2014 in 4 FEMA regions, 33 of which assessed at least one, if not two or all three, of the core capabilities we identified as proxies for NIMS elements—operational coordination, operational communication, and public information and warning. Of the 35 reports we reviewed, 30 assessed operational coordination, 22 assessed operational communication, and 12 assessed public information and warning. Generally, for the capabilities assessed, the AARs identified strengths as well as issues and challenges in performing related activities and tasks. Specifically, 24 of the 30 exercises that assessed operational coordination, 21 of the 22 exercises that assessed operational communication, and 8 of the 12 exercises that assessed public information and warning identified areas for improvement with those capabilities. While exercises are meant to assess the ability to meet exercise objectives by documenting strengths, areas for improvement, core capability performance, and corrective actions in an AAR, some of the AARs documented problems with NIMS-related exercise objectives, indicating that there is room for improvement in NIMS implementation. For example, four of the AARs found that an Incident Command System (ICS) was not established among responding agencies, even though NIMS calls for using such a structure. Similarly, another AAR concluded that responding staff were unfamiliar with their position-specific roles and responsibilities, even though NIMS calls for emergency management and response personnel to have a clear understanding of their roles and responsibilities. The report noted that the staff displayed difficulty in monitoring, gathering, receiving, organizing, and sharing incident information—all activities that implementation of NIMS is designed to facilitate. Another exercise evaluating operational communication found that there was no communication among the different law enforcement agencies and officers during the exercise. These examples demonstrate that AARs can provide a basis for FEMA to develop a more comprehensive and sophisticated understanding of states’ NIMS implementation, and suggests that there is room for improvement in NIMS implementation for some jurisdictions.

NIC officials said they do not verify the self-reported NIMS implementation information that states submit because of the scope and breadth of the information. However, in reviewing AARs to help inform FEMA’s understanding of NIMS implementation, FEMA regional office staff could be in a better position to assess how well states have actually implemented NIMS, as regional staff have more direct responsibilities related to states’ NIMS implementation. Specifically, FPCs—who lead regional efforts to implement the National Preparedness System across the federal, state, tribal, and local jurisdictional levels—are responsible for monitoring NIMS compliance and implementation. FPCs are also responsible for assisting in planning, design, execution, and evaluation of exercises. In addition, each regional office has a NIMS coordinator who is responsible for working with states to provide NIMS information and technical assistance to support, facilitate, and enhance their implementation of NIMS. One FPC stated that officials in his region review state AARs within the region to identify needed corrective actions that could be taken by states. Similarly, one regional NIMS coordinator developed a checklist to perform NIMS implementation state monitoring visits and said he visits 2 states each year in his region to ask questions about implementation of various NIMS principles and provide assistance. However, while some of these officials have taken independent steps to assess some level of NIMS implementation, the regional offices we spoke with do not systematically assess states’ level of NIMS implementation by, for example, reviewing AARs because there are no policies or procedures that call for these activities. Specifically, while states are to submit AARs to FEMA’s National Exercises Division, and to the regional EMPG Program Manager, as a requirement for receiving EMPG grants, FPCs and NIMS coordinators are not required to review them to provide validation of the NIC’s NIMS implementation assessment.

FEMA regional office officials we spoke with, along with several state emergency management officials, agreed that assessing NIMS implementation using AARs could provide FEMA a way to further assess and elaborate on the information states and territories currently provide. For example, the National Exercise Division, another division within the National Preparedness Directorate, conducted an assessment of NIMS in 2013 by reviewing AARs and found that failures in NIMS performance were predominantly due to failures in implementation of NIMS. The report recommended, among other things, that the agency modify existing training and exercises to allow responders to integrate NIMS concepts and improve assessments of the ICS (as a component of NIMS performance) by incorporating relevant principles as criteria of performance. According to a NIC official, FEMA has been working on the recommendations and expects to implement the actions when FEMA releases an update to the NIMS doctrine. Finally, while the National Exercises Division did assess NIMS implementation issues in 2013 by analyzing AARs, regional offices and the NIC do not systematically or regularly assess NIMS implementation. Using AARs to systematically assess NIMS implementation over time could allow the regions and the NIC to identify recurring issues in jurisdictions.

HSPD-5 calls for FEMA to (1) establish a mechanism for ensuring ongoing management and maintenance of the NIMS, including regular consultation with other federal departments and agencies and with state and local governments, and (2) develop standards and guidelines for determining whether a state or local entity has adopted NIMS. In addition, OMB Circular No. A-11 states that data limitations can lead to bad decisions resulting in lower performance or inaccurate performance assessments, and data limitations can include imprecise measurement and recordings, incomplete data, and inconsistencies in data collection procedures. Consistent verification and validation of performance data support the general accuracy and reliability of performance information, reduce the risk of inaccurate performance data, and provide a sufficient level of confidence to Congress and the public that the information presented is credible as appropriate to its intended use. As we have previously reported, FEMA’s reliance on self-reported data from states and lack of verification for FEMA preparedness grants has presented reliability concerns. While AARs are also self-reported, they use a narrative format that includes analyses of the various aspects of actions following an exercise or actual event. State officials from two states we spoke with said that they review AARs within their state in order to evaluate NIMS implementation. For example, officials from one state said they use AARs to identify gaps in NIMS implementation so they can align resources to support better implementation. This is because exercises enable stakeholders to identify both capability gaps and areas for improvement. Further, the purpose of exercises is to provide a low-risk environment to test capabilities, familiarize personnel with roles and responsibilities, and foster meaningful interaction and communication across organizations. As such, evaluating AARs could provide an additional opportunity to look for areas of NIMS implementation that could be improved.

According to NIC officials, they meet the requirements of HSPD-5 through ongoing discussions with NIMS stakeholders, including state emergency managers, to ensure ongoing management and maintenance of NIMS, and by obtaining self-assessment information provided by states on NIMS implementation in the URT that provides standards and guidelines. Additionally, in order to meet the requirements of HSPD-5, in 2006 the NIC developed implementation guidance through the NIMS Compliance Objectives and Metrics for states and territories, tribal nations, and local governments, which was last updated in 2010. However, NIC officials said that, given the large number of states and their jurisdictions that have adopted NIMS, any efforts to validate the information states provide on their implementation efforts would be prohibitively difficult and costly. While verifying each state and jurisdiction’s self-reported information may not be feasible, states are already required to develop AARs under the EPMG program, which could provide a source in addition to the URT by which FEMA could assess progress made on implementation of NIMS, as the 2013 assessment by the National Exercise Division demonstrates. States are already required to develop AARs following their preparedness exercises, based on grant program requirements. Moreover, FEMA’s regional offices have multiple positions with responsibilities for NIMS implementation, as well as preparedness exercises. Therefore, developing policies and procedures for regional office staff to review and assess information on states’ NIMS implementation, gleaned through a review of exercise AARs, would not only help FPCs and NIMS coordinators better meet their responsibilities for supporting and enhancing regional preparedness, but also provide the NIC a means, in addition to the URT, to further evaluate states’ self-reported assessments of NIMS implementation.

Additional Opportunities Exist for FEMA Regional Offices to Enhance Collaboration with Regional Advisory Council Members

RAC Members and Liaisons View RAC Meetings as Useful and Well Run

As described earlier, the Post-Katrina Act required that each FEMA region establish a Regional Advisory Council (RAC) in order to provide the types of advice on emergency preparedness in their regions listed in the Post-Katrina Act to the FEMA regional administrators. RAC members who responded to our survey reported that they found RAC meetings to be a useful collaborative experience, allowing them to obtain good information, have interactions with other RAC members and FEMA officials, and to offer advice to FEMA on regional emergency-related issues. For example:

About 90 percent said that the RAC meetings were moderately or very effective in providing information on how to make the region better prepared for emergencies; the information communicated at RAC meetings was moderately or very effective (89 percent); coordination between the RAC and the regional office was moderately or very effective (89 percent); the quality of the communications from their FEMA regional office was good or excellent (91 percent); their regional office was effective in asking members about their concerns or suggestions about emergency preparedness issues in their region (93 percent); and their participation was encouraged (93 percent).

About 81 percent said that the meetings provide a useful forum in which to offer advice to FEMA on regional emergency-related issues; most said that they likely would serve another term, if invited (81 percent); the overall preparedness of the region increased moderately or greatly as a result of RAC meetings (69 percent); and, in general, the meetings were well run, on time, on topic, and relevant to regional emergency-related issues (89 percent).

Many of those FEMA regional offices that convene RAC meetings (see below for a discussion of the variations in the occurrence of meetings) prepare and send out agendas in advance of the meetings and collaboratively involve RAC members in planning. Conducting early outreach to participants and stakeholders to identify shared interests is an effective collaborative implementation approach identified in GAO’s Implementation Approaches Used to Enhance Collaboration in Interagency Groups. RAC members who responded to our survey reported that they appreciated the information provided to prepare them for the RAC meetings. For example, 96 percent reported that they were notified in advance about the meetings, 86 percent stated that the agendas provided information necessary for preparing for the RAC meetings, and 79 percent believed that distributing the minutes after the meeting was a moderately or very effective approach.

Some FEMA Regions Do Not Routinely Meet with RACs to Collaborate on Emergency Preparedness

Although the RAC charters adopted by each of the 10 regions require that their RACs meet twice annually, some RACs have not met routinely in recent years to collaborate with FEMA regional officials to provide advice and input from regional stakeholders. The DHS Inspector General reviewed the FEMA Region IX Office in 2012 and the Region V Office in 2015 and concluded that because the RACs in those regions had not met for a period of years, FEMA’s regional offices could be “missing opportunities to identify and remediate weaknesses or deficiencies in preparedness, protection, response, recovery, and mitigation activities.” Figure 2 summarizes the number of RAC meetings in each region since 2007 through August 2015 (including both in-person meetings and meetings conducted as interactive webinars).

One region cited the fact that they did not have funds to reimburse RAC members for travel expenses, and another cited travel costs as a problem for RAC members in a region as geographically large as theirs. Another region stated that while they paid for the travel expenses, this necessarily had to come from their available funds and that therefore, it meant having less for other potential needs.

In our survey of RAC members, respondents cited several reasons why they did not attend RAC meetings. For example, 68 percent of respondents cited scheduling commitments and conflicts, 26 percent cited insufficient or no funding for travel, and 11 percent said that the meeting locations were inconvenient. Some RAC members said that their RAC should definitely or probably meet more frequently (41 percent), while others did not (30 percent said “probably” or “definitely not” and 23 percent said they were unsure or had no opinion). In open-ended comments on RAC meetings attended by RAC members, 6 RAC members stated that they would prefer having more in-person meetings versus teleconferences or videoconferences.

FEMA Region V, in which the RAC did not meet for several years, took steps starting in August 2014 to address a lack of interest from the majority of the members. For example, regional officials invited RAC members to other regional meetings with other federal agencies—a process used by some other regions. In 2015, the regional office resumed having RAC meetings, with one taking place in mid-January with 11 of 12 members attending. According to the regional RAC liaison, the high attendance was the result of the regional administrator having personally reached out to the individuals who had been recommended for RAC membership and inviting them to participate. He said this individualized communication with each person on the team helped to instill interest and motivate involvement in the meeting, in the view of the liaison, and that the personal involvement of the regional administrator makes the RAC more effective.

The Post-Katrina Act requires FEMA regional offices to establish RACs in order to provide input to each regional administrator on regional emergency management issues by identifying any geographic, demographic, or other characteristics peculiar to any state, local, or tribal government within the region that might make preparedness, protection, response, recovery, or mitigation more complicated or difficult, among other things. Additionally, GAO’s Key Considerations for Implementing Interagency Collaborative Mechanisms identified meetings that bring stakeholders together as an effective collaborative practice, noting particularly that such relationship-building is vital in responding to disasters.

By routinely obtaining input from RAC members—whether through in- person meetings or via remote connection or other means—FEMA regional offices could better ensure they are identifying geographic, demographic, or other issues within the region to enhance emergency preparedness. Additionally, for those FEMA regional offices that face participation challenges, assessing the reasons for these barriers and identifying targeted solutions, such as consistently offering remote participation, or, as used by one region, direct communication from the regional administrator to RAC members, could better ensure they are able to obtain such input from the RACs effectively and on a regular basis.

FEMA Regional Offices Are Responsive to RAC Suggestions but Do Not Regularly Provide Feedback on Their Recommendations

RAC members who responded to our survey generally indicated that FEMA regions were responsive to their suggestions.

Seventy-five percent of the RAC members said that they had made suggestions concerning issues related to emergency management at RAC meetings or between RAC meetings; 79 percent stated that their overall sense that FEMA was listening to their concerns increased moderately or greatly during the time on which they had served on their RAC; and 74 percent stated that FEMA was acting to address their concerns.

Sixty-seven percent of the respondents said FEMA had taken action to a moderate, great, or very great extent in response to suggestions made by RAC members.

FEMA RAC liaisons we contacted by e-mail reported having taken action in response to RAC member suggestions. Specifically, 9 of the 10 FEMA RAC liaisons reported that input from their RACs had either led to changes in specific FEMA policies or led to generally better informing FEMA about members’ concerns. For example, at the suggestion of their RACs, regional offices created a regional training and exercise coordination website to share information; forwarded a suggestion from a RAC to revise FEMA national policy to include ice storms as a reimbursable emergency; and developed and distributed monthly preparedness themes to help synchronize federal, state, and local officials’ focus on common topics, among other things.

Although some FEMA regional offices have incorporated various recommendations made by RAC members, 50 percent of the respondents in our survey of RAC members said that FEMA should provide more detailed feedback to them on suggestions or advice they made at RAC meetings. Of the 10 regional offices, 7 reported that they tracked outcomes of RAC efforts; 3 said they posted suggestions on FEMA’s internal website (which is not accessible to non-FEMA employees, including RAC members), and 4 said they used e-mail to forward meeting notes to RAC members. One RAC member from our survey commented on the need for a tracking system for the status and resolution of issues raised, and another RAC member commented on the need for better follow-up on recommendations. Developing a mechanism, such as a Listserve or a dedicated website, to periodically notify all RAC members in all 10 regions about the status of suggestions made by RACs to regional administrators or to FEMA could help FEMA keep RAC members aware of the status of their recommendations and thereby address the concerns of those who wanted more detailed feedback.

In March 2014, FEMA’s NAC identified the need for FEMA to re-examine the outcomes of its existing regional committees and councils, including the RACs. Reporting the outcomes of RACs’ suggestions to the NAC could help FEMA address this identified need. The NAC’s recommendation to FEMA also reflects the key considerations for implementing interagency collaborative mechanisms found in GAO’s Implementation Approaches Used to Enhance Collaboration in Interagency Groups. As the report stated, many of the meaningful results that the federal government seeks to achieve, such as national preparedness, require the coordinated efforts of more than one federal agency, level of government, or sector. Implementation approaches for interagency collaborative mechanisms include developing a plan to communicate outcomes and track progress, and developing methods to report on the group’s progress that are open and transparent. As noted previously, while some regions have taken steps to track outcomes of RAC efforts, none of the regional offices has a process for regularly providing feedback on the status of recommendations. Providing feedback to RAC members on their suggestions could enhance FEMA regional offices’ collaboration with their RACs.

Although Many RAC Members Say Their Concerns Are Communicated to the NAC, Others Want More Feedback on Responses to Suggestions

The NAC, established by the Post-Katrina Act, incorporates state, local, and tribal governments, nonprofit, and private sector input with regard to improving emergency preparedness and response to natural and manmade emergencies. Although almost two-thirds of RAC members who responded to our survey said they believed their concerns were communicated by FEMA to the NAC, some RAC members commented that they wanted more feedback on the NAC’s responses to suggestions. However, FEMA has not established policies and procedures to communicate RACs’ regional concerns directly to their national-level counterparts on the NAC, as proposed in 2011. Almost two-thirds of respondents said FEMA was (very or moderately) effective in communicating RAC concerns and issues to the NAC. However, in open ended comments, six RAC members stated they that would like to receive more information about the NAC and what it was addressing.

In September 2010, a FEMA local, state, tribal, and federal task force for improving national emergency preparedness recommended that communications between the RACs and the NAC be more robust. Specifically, the task force recommended that RACs receive policy information briefings directly from the NAC and provide input directly to the NAC in order to provide local, state, tribal, and territorial officials with influence across stages of the preparedness policy process and to create an informational exchange between the NAC and the RACs to make local, state, tribal, and territorial officials fuller partners in the preparedness policy process.

In June 2011, FEMA responded to the recommendation from the task force with an implementation plan, stating that, by November 24, 2011, it would revise its policies and procedures to systematically link the RACs to each other, the NAC, and FEMA headquarters, to help RACs influence national emergency management policy. FEMA officials said they held a meeting of all the RAC regional liaisons in August 2011 to propose options to create a “NAC-RAC interface,” among other things. However, they did not identify any additional subsequent actions they had taken to implement the plan. For example, although FEMA stated that the August 2011 RAC liaison meeting had resulted in adding a “RAC Report Out” by a regional administrator as a standing agenda item for the NAC meetings, none of the NAC meeting agendas for any NAC meeting between August 2011 and September 2015 showed “RAC Report Out” as an agenda item. In addition, our review of the minutes for all the NAC meetings held between August 2011 and September 2015 found only an occasional mention of issues of concern to RACs reported on by FEMA regional administrators or others present.

Officials from FEMA’s Office of Regional Operations said that individual RACs can transmit their concerns to the NAC, such as by letter or through FEMA. The FEMA Office of the NAC provided a detailed summary of the status of recommendations made by the NAC to FEMA since 2008 to NAC members at the September 2015 NAC meeting. However, this summary did not include the status of recommendations or suggestions or communications made by RACs to the NAC. As such, these processes do not appear to provide a systematic way in which to ensure that issues raised in RAC meetings are consistently brought to the attention of the NAC for its consideration or to provide a consistent feedback mechanism for RAC members.

In November 2015, FEMA’s Office of the NAC said that although it is precluded from establishing a formal mechanism between the RACs and the NAC, it is working with FEMA’s Office of Regional Operations to informally enhance coordination between the NAC and the RACs, such as distributing NAC recommendation memoranda with the RACs for informational purposes. Establishing policies and procedures, as suggested by FEMA’s task force and 2011 implementation plan, to enhance coordination and communication between the RACs and the NAC could create greater opportunities for regional insights about preparedness to inform national preparedness policies and a two-way informational exchange between the NAC and the RACs to make local, state, tribal, and territorial officials fuller partners in the preparedness policy process.

Conclusions

Because of the key role FEMA’s regional offices play in national disaster preparedness efforts, effective regional coordination and collaboration between FEMA headquarters, FEMA regions, and state and local stakeholders is essential. Multiple external and internal assessments and our work have identified challenges, such as lack of coordination in grant- monitoring visits and inconsistent guidance, in FEMA’s grant management. These challenges continue to hamper the effectiveness of interactions between FEMA and state officials in implementing the preparedness grants program. A plan with time frames, goals, metrics, and milestones for addressing these challenges would better enable GPD officials to monitor and improve their efforts to resolve long-standing problems in coordination of monitoring activities and the consistency of guidance given to state grantees. Regarding oversight of FEMA’s efforts to implement the national system for managing incidents, there are opportunities to access more sophisticated and comprehensive information using FEMA regional staff to better assess NIMS implementation. Further, using AARs that are already required to be developed and submitted could provide a better opportunity to assess NIMS implementation, including areas of success as well as areas where improvement is needed. However, without policies and procedures that call for systematically evaluating these reports, regional offices and the NIC are missing a key opportunity to better assess national NIMS implementation, which could enhance regional preparedness. Finally, FEMA regional offices could more consistently and systematically take steps to fully leverage RACs that can provide important inputs to national preparedness efforts. For example, by ensuring that RACs provide feedback to RAC members on the status of their recommendations, FEMA could enhance RACs’ collaborative efforts and satisfaction of RAC members. Similarly, by enhancing RACs’ connectivity with the NAC, FEMA could increase the potential value of regional coordination and collaboration and the resulting contributions to regional preparedness efforts.

Recommendations for Executive Action

To promote more effective grant management coordination, the Secretary of Homeland Security should direct the FEMA Administrator to develop a plan with time frames, goals, metrics and milestones detailing how GPD intends to resolve longstanding challenges associated with its existing hybrid grants management model, which divides responsibilities between regional and headquarters staff.

To enable more sophisticated and comprehensive awareness of states’ NIMS implementation, the Secretary of Homeland Security should direct the FEMA Administrator to develop policies and procedures for regional staff to review AARs from preparedness exercises within their region, and headquarters staff to review these evaluations in order to have a better understanding of NIMS implementation.

To enhance the value of RACs, the Secretary of Homeland Security should direct the FEMA Administrator to take the following three actions: 1. Ensure that all regional offices routinely obtain input—whether in person, by teleconference or by other remote connection—from their RAC members on ways to enhance overall emergency preparedness in their regions. In cases where RAC member participation is low, regional offices should assess and identify targeted solutions for increasing member participation, such as offering remote participation or alternative forums; 2. Develop a mechanism to update RAC members on the status of recommendations made by RACs to FEMA; and 3. Establish processes for enhanced coordination and communication between the RACs and the NAC.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for comment. DHS provided technical comments, which we incorporated as appropriate. On January 15, 2016, DHS also provided written comments, reproduced in full in appendix IV. DHS concurred with four of our recommendations and did not concur with one.

DHS did not concur with our first recommendation that the Secretary of Homeland Security direct the FEMA administrator to develop a plan with time frames, goals, metrics, and milestones detailing how GPD intends to resolve longstanding challenges associated with its existing hybrid grants management model. In its comments, DHS stated that it disagreed with GAO’s characterization of longstanding challenges in managing preparedness grants. As we stated in the report, multiple assessments dating back to 2009 have reported challenges with the hybrid model that splits management of preparedness grants between FEMA’s headquarters and regional offices. As also noted in the report, officials from 4 FEMA regional offices and officials from 3 states within those regions provided numerous examples of a lack of coordination between headquarters and regional staff in managing preparedness grants, including instances that took place in 2014 and as recently as September 2015. Based on our review of the past assessments and the audit work we performed, we believe that these long-standing challenges have yet to be resolved.

In its written comments, DHS also stated that FEMA already empowers its ten regions to coordinate and communicate with headquarters and other regional offices to facilitate working relationships. While FEMA has taken various actions since 2008 to improve coordination related to preparedness grants—some actions as recent as February 2015—these steps have not resolved these challenges, as described in the report.

When we spoke with GPD officials in September 2015, they agreed that there were continuing opportunities to improve coordination and communications in their management of preparedness grants. Therefore, we continue to believe that FEMA would benefit from a more strategic approach that a plan, with time frames, goals, metrics, and milestones detailing how officials intend to resolve longstanding challenges associated with the existing hybrid model, could provide.

DHS also expressed concern about the wording of our recommendation and how the recommendation, as worded, could be closed. Specifically, DHS stated that FEMA is not aware of nor did the recommendation provide any specific criteria that could be used to gauge the success of any proposed plan. While we recommended that DHS develop a plan with time frames, goals, metrics, and milestones for resolving its longstanding coordination challenges, we did not prescribe specific actions to address these challenges. However, our work across the federal government has identified implementation approaches used to enhance collaboration that could inform GPD officials’ efforts to clearly define short-term and long- term outcomes and track and monitor their progress. Nevertheless, FEMA could implement this recommendation by developing a plan and we believe that doing so would address the intent of this recommendation.

Regarding our second recommendation that the Secretary of Homeland Security direct the FEMA Administrator to develop policies and procedures for regional staff to review AARs from preparedness exercises within their region, and headquarters staff to review these evaluations in order to have a better understanding of NIMS implementation, DHS concurred and said it recognizes that joint efforts by regional and headquarters staff are essential to enable more sophisticated and comprehensive awareness of NIMS implementation. However, DHS expressed concerns about overstating the value of AARs as a means to improve understanding of NIMS implementation. Specifically, DHS stated that FEMA does not agree that the “best way to assess how well NIMS has been implemented is by assessing states’ performance in preparedness exercises and real-world events.” As noted in the report, most all of the officials we spoke with believed that the current method of determining NIMS implementation—states self- certification via the URT—was not an effective method of determining NIMS implementation, and officials in all of the FEMA regions and 8 of the states we interviewed said that looking at how effectively a jurisdiction responds during an exercise or real world event is the best way to assess NIMS implementation. DHS also stated that the extent to which a jurisdiction has implemented NIMS is nuanced and therefore impossible to assess accurately from an AAR. The report does not suggest that reviewing AARs would give a comprehensive and definitive assessment of NIMS implementation but rather that it could provide FEMA an additional way to assess and elaborate on the information states currently provide. In fact, FEMA used AARs to conduct an assessment of NIMS in 2013, which resulted in some recommendations to help improve NIMS implementation. Finally, as we have previously reported, the Post-Katrina Act requires FEMA to carry out the National Exercise Program. The Act established the Program, among other things, as a means for FEMA to test and evaluate NIMS implementation efforts. As a result, we continue to believe that AARs, an essential part of the National Exercise Program, are an important part of FEMA’s efforts to assess NIMS implementation and we added a reference to this requirement to our final report. DHS did not identify any specific steps to address this recommendation but said that the National Preparedness Directorate would work with regional NIMS coordinators and the states to explore options during calendar year 2016, with potential changes to be implemented in calendar year 2017.

In its written comments, DHS concurred with our third, fourth, and fifth recommendations related to enhancing the value of RACs. Specifically, regarding the third recommendation that the Secretary of Homeland Security direct the FEMA administrator to ensure that regional offices routinely obtain input from their RAC members, DHS concurred and stated that the regions will conduct two meetings a year with their RAC members; that a RAC coordinator within each region will be responsible for the oversight of these meetings; and that open dialogue will be encouraged between RAC members and the regions, with each region establishing a RAC coordinator to facilitate this communication. If implemented as planned, these actions should help address the intent of the recommendation to ensure that regional offices routinely obtain input from their RAC.

Regarding the fourth recommendation that the Secretary of Homeland Security direct the FEMA administrator to develop a mechanism to update RAC members on the status of recommendations made by RACs to FEMA, DHS concurred and stated that it will build into each RAC meeting a briefing by regional personnel on the status of recommendations, if any, made during the previous meeting. The regional administrator or RAC coordinator within the region will be responsible for the brief. Additionally, if RAC members are not satisfied with the implementation of RAC recommendations, it will be expected that the regional office will provide information on why a specific recommendation was not adopted or why implementation is delayed. DHS further stated that each region may have its own method of dispensing updates to its RAC members between meetings, including monthly email updates, posting on their website, or informational calls. If implemented as planned, these actions should help address the intent of the recommendation to update RAC members on the status of its recommendations to FEMA.

Regarding our fifth recommendation that the Secretary of Homeland Security direct the FEMA administrator to establish processes for enhanced coordination and communication between the RACs and the NAC, DHS concurred and stated that regional administrators will continue to attend NAC meetings and provide FEMA regional updates that include RAC activities with the regions. Moreover, regional coordinators will ensure that RAC members are aware of any NAC meeting occurring in their own region, and that the RAC sharepoint site will be used as a platform to update RAC coordinators on impending NAC meetings to share with their RAC members. If implemented as planned, these actions should help address the intent of the recommendation to establish processes for enhanced coordination and communication between the RACs and the NAC.

We are sending copies of this report to Secretary of Homeland Security, appropriate congressional committees, and other interested parties. This report is also available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (404) 679-1875 or at CurrieC@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of this study were to assess the extent to which the Federal Emergency Management Agency (FEMA) headquarters and regional offices have (1) addressed grant management coordination challenges between headquarters and the regions; (2) established a comprehensive system to assess National Incident Management System (NIMS) implementation; and (3) collaborated with Regional Advisory Council (RAC) members.

To address the first objective, we gathered and reviewed relevant documentation, such as FEMA assessments of grants managed by Grant Programs Directorate (GPD), FEMA’s previous delegations of grant management responsibilities to regional offices, agency task force reports on advantages and disadvantages of regionalization, FEMA headquarters and regional office grant management roles and responsibilities, FEMA documentation on regionalization pilot programs, and other memoranda and internal documents. We also reviewed prior GAO reports on regional preparedness, grant management, and consolidation of management functions. We interviewed FEMA’s Director of the Preparedness Grants Division within GPD and other senior GPD officials to discuss programmatic grant management as well as previous FEMA assessments of the impact of moving management responsibilities to FEMA regional offices. We also interviewed grant management officials from 4 FEMA regional offices—Regions III, IV, VI, and IX—to discuss financial grant management and coordination of monitoring activities with GPD. These offices were selected in order to provide a mix of geographic locations, types of disasters typically experienced, and whether the regional office had been selected to participate in FEMA grant management pilot programs. Similarly, we interviewed officials from 9 states or territories (Arkansas, California, Florida, Georgia, Louisiana, Nevada, Texas, Virginia, and Washington, D.C.) from the 4 FEMA regions to discuss grant management within the states as well as interactions with FEMA grant management officials. These states were selected to reflect a diversity of experiences within the 4 FEMA regions we chose. While the information gained from these interviews cannot be generalized across all states, it provides useful insights into the nature of FEMA GPD and regional coordination with states with regard to grant management.

To address the second objective, we reviewed NIMS-related documents such as the NIMS Doctrine, NIMS Compliance Guidance, NIMS implementation reports, and results of the NIMS implementation questions in the Unified Reporting Tool (URT). We also reviewed documents on the Homeland Security Exercise and Evaluation Program (HSEEP) and on preparedness grants requirements. We compared FEMA’s efforts in NIMS implementation and verification with Homeland Security Presidential Directive-5 and Office of Management and Budget (OMB) Circular No. A-11, regarding data validation and verification. We interviewed officials at FEMA’s National Preparedness Directorate, which includes the National Integration Center (NIC)—the office responsible for NIMS implementation—to discuss the NIC’s efforts in implementing NIMS. To discuss NIMS implementation efforts at the regional and state levels, interactions between regional and state officials, and mecha nisms for verifying how well NIMS is being implemented, we also interviewed officials from the 4 FEMA regional offices and the 9 states or territories mentioned above. In addition, because of the recommendations of officials in several regional offices and states, we interviewed an official from a 10th state (Oklahoma) from within the selected FEMA regions. Again, while the information gained from these interviews cannot be generalized across all states, it provided useful insights into the various efforts in implementing NIMS. Additionally, we analyzed all 35 full-scale exercise after-action reports (AAR) that FEMA received for exercises conducted during fiscal year 2014 and in the 4 regions we visited. We used the AARs as they provide qualitative analysis of outcomes of exercises, which are conducted in order to identify capability gaps and areas for improvement. The purpose of the analysis was to determine if they could be used to assess the level of participants’ implementation of NIMS. We did this by developing a crosswalk that linked the six NIMS elements listed in the URT, and that states should incorporate into exercises, to a core capability—the unit by which exercises are evaluated. In developing this crosswalk, we found three core capabilities that we used as proxies for the six NIMS elements that are to be incorporated into exercises—operational coordination, operational communication, and public information and warning. While aspects of NIMS implementation could be gleaned from other core capabilities, for the purposes of this analysis we used the three core capabilities identified through our crosswalk. Of the 35 reports we analyzed, 33 assessed at least one, if not two or all three, of the core capabilities we identified as proxies. In those reports where one or more of the core capabilities were addressed, we looked to see if there were issues associated with those core capabilities, and the nature and extent of the issues, by reviewing various sections of the reports. For the core capability of operational coordination, we also looked at the exercise objectives to see if they aligned with some element of NIMS components. If an AAR reported that the core capability that we assessed either was performed with some challenges, was performed with major challenges, or was unable to be performed, we included it in our count of a capability that encountered some issues. From this we could glean the extent to which various NIMS elements are working well and those that still have room for improvement. While results of these exercises are not generalizable to the country as a whole, they do provide useful insights into NIMS implementation in some areas.

To address the third objective, we interviewed FEMA headquarters and regional office officials and analyzed FEMA and Department of Homeland Security (DHS) Office of Inspector General (OIG) reports with information and findings relevant to RACs. We attended two RAC meetings in 2 regions that took place during the early part of our review, and obtained documentation of agendas and other relevant materials for RAC meetings that occurred between 2007 and 2014 for the RACs in all 10 FEMA regions. We also reviewed agendas and meeting minutes for NAC meetings that took place between August 2011 and September 2015. We reviewed relevant federal laws and regulations, federal internal controls standards, and leading practices for program management.

To identify the extent to which RAC members view their RAC experiences as effective in conveying their concerns to FEMA, we conducted a self- administered, web-based questionnaire survey of RAC members of each FEMA region’s RAC; FEMA’s 10 regional offices identified an initial population of 133 active RAC members. (See app. II for a copy of the survey questionnaire and the results.)

Using e-mail addresses provided by each of the FEMA regions for their RAC members (which included the RAC members’ names and, in most cases, their place of work or other occupational identifier), we e-mailed each RAC member a link to a secure survey website, along with a unique identifier and password to control access to each member’s questionnaire. Most survey questions were closed-ended, in which RAC members selected from a list of possible responses. To obtain additional narrative and supporting context, survey respondents were given opportunities to provide additional open-ended comments throughout the survey.

The survey began on May 27, 2015, and data collection ended on June 29, 2015. We e-mailed the survey link to 126 of the RAC members identified by FEMA regional offices. Of these 126 e-mail contacts, e-mails to 14 were initially undeliverable and 2 were e-mails for FEMA employees who had been incorrectly listed as RAC members (FEMA employees cannot be RAC members). Of the 14 non-FEMA RAC members, we determined that 12 were no longer RAC members. Of the remaining 2, one informed us that she had only recently been appointed to a RAC and had not attended any RAC meetings, and the other told us in a follow-up telephone call that he had not been a RAC member for at least 4 years. On the basis of this information about RAC membership status, we defined all 16 cases as ineligible and excluded them from the survey population. Of the 110 contacted that were still presumed to be eligible members, 77 responded to the survey, resulting in a 70 percent response rate. Of the 77 respondents, 3 answered an initial screening question stating that they were not currently members of the RAC in their region, which left a total of 74 respondents whose responses were usable for our analysis of the rest of the questions. During the course of survey data collection, we sent periodic reminder e-mails to all nonrespondents to encourage participation in the survey. We also conducted tailored follow- ups by e-mail or telephone with selected nonrespondents, for example, those only partially completing their questionnaires.

Because our survey was designed to include the entire population of RAC members, our results are not subject to sampling error. However, the practical difficulties of conducting any questionnaire survey may introduce a variety of non-sampling errors. For example, differences in how a particular question is interpreted, the sources of information available to respondents, or the types of people who do not respond to the survey or a particular question can introduce error into survey results. We included steps in the survey design, data collection, and analysis stages to minimize these types of non-sampling errors. To minimize measurement error and nonresponse error, we designed draft questionnaires in collaboration with GAO survey specialists. We conducted pretests by telephone with three RAC members from three regions. On the basis of survey specialist input and these pretests, we made revisions to questionnaire drafts as necessary to reduce the likelihood of measurement and nonresponse errors (the types of nonresponse associated with the perceived burden, lack of question clarity or relevance to the respondent). In addition, our analysts answered respondent questions and resolved difficulties that respondents had in completing our questionnaire. We asked open-ended comment questions at various points in the questionnaire to allow respondents to explain or provide context for their answers, which helped inform and corroborate our interpretation and analysis of the survey results. We examined survey results to identify and remediate any inconsistencies or other indications of response error. Finally, we made multiple follow-up contacts with non- respondents throughout data collection to reduce nonresponse. To minimize the possibility of data processing error, a second data analyst independently verified the accuracy of all computer analyses. In addition, RAC members made their responses directly into an automated web survey instrument, preventing errors associated with manual data entry of written answers.

We conducted our work from October 2014 to February 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Results of GAO Survey of Federal Emergency Management Agency (FEMA) Regional Advisory Council Members

Below is the text of the survey as it was sent to Regional Advisory Council (RAC) members. Following the cover page, the questions are shown, along with the results in percentages. Narrative answers to open-ended text questions are not displayed to prevent the identification of individual respondents. Percentages may not always sum to 100 percent because of rounding of decimal figures.

Appendix III: National Incident Management System (NIMS) Implementation Questions Asked of States and Territories in the Unified Reporting Tool

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Chris Keisling (Assistant Director), David Alexander, Chuck Bausell, David Dornisch, Eric Hauswirth, Susan Hsu, Valerie Kasindi, Stuart Kaufman, Tracey King, Marvin McGill, Carl Ramirez, Robert Rivas, and Jonathan Tumin made key contributions to this report.