Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA's Risk-Based Grant Assessment Model
Released: 2018-09-06T00:00:00

Letter

Background

DHS’s Homeland Security Grant Program

The federal government has provided financial assistance to public and private stakeholders for preparedness activities through various grant programs administered by DHS through its component agency, FEMA. Through these grant programs, DHS has sought to enhance the capacity of states, localities, and other entities, such as ports or transit agencies, to prevent, prepare for, protect against, respond to, and recover from, and mitigate a natural or manmade disaster, including terrorist incidents. Two of the largest preparedness grant programs are the SHSP and UASI grant programs.

SHSP grants provide federal assistance to support states’ implementation of homeland security strategies to address the identified planning, organization, equipment, training, and exercise needs at the state and local levels to prevent, prepare for, protect against, and respond to acts of terrorism. SHSP grants are annually awarded to all the nation’s 56 states and territories. SHSP grant awards are calculated in two parts. All states and territories are to receive a minimum grant amount required by law, based on a percentage of the total amount of SHSP and UASI appropriations in a given fiscal year. The remaining award amounts are based on FEMA’s risk-based grant assessment model.

UASI grants provide federal assistance to address the unique needs of high-threat, high-density urban areas, and assists the areas in building an enhanced and sustainable capacity to prevent, prepare for, protect against, respond to acts of terrorism. Since 2015, Congress has instructed through the Explanatory Statements accompanying the annual DHS Appropriations Acts that the UASI grants should be awarded to urban areas that reflect up to 85 percent of nationwide risk. For the UASI program, FEMA uses the risk-based grant assessment model each year to identify those urban areas that will be eligible to receive funding.

Annual funding for the SHSP and UASI programs have generally declined over the period of fiscal years 2008 through 2018, but have remained consistent since fiscal year 2016. Figure 1 shows the changes to SHSP and UASI programs’ annual funding during this period. For example, annual funding for SHSP decreased from about $861 million in fiscal year 2008, to $402 million in fiscal year 2018. During this same period, annual funding for UASI also declined, from about $782 million in fiscal year 2008 to $580 million in fiscal year 2018. However, annual funding for the UASI program has been higher than the SHSP program since fiscal year 2010.

FEMA’s Risk-based Grant Assessment Model for Distributing Funding Awards

Risk=Threat x Vulnerability x Consequence Threat–A natural or man-made occurrence, individual, entity, or action that has or indicates the potential to harm life, information, operations, and/ or property Vulnerability–Physical feature operational attribute that renders an entity, asset, system, network, or geographic area open to exploitation or susceptible to a given hazard. Consequence–Effect of an event, incident, or occurrence, commonly measured in four ways: human, economic, mission, and psychological, but may also include other factors such as impact on the environment.

FEMA’s risk-based grant assessment model uses three variables: Threat, Vulnerability, and Consequence. The purpose of this model is to apply a risk management process to provide a structured means of making informed trade-offs and choices about how to use finite resources effectively, and monitoring the effect of those choices. Specifically, inherent “uncertainty” is associated with any effort to develop a risk model such as assessing the risk of terrorist attacks, and thus, requires the application of policy judgments and analytic assumptions. The effect that uncertainty has on the results of the risk model can be especially important if the model produces materially different results in response to even small changes in assumptions, often referred to as the “sensitivity” or “robustness” of a model’s assumptions and results.

As we reported in June 2008, FEMA’s risk-based grant methodology and its continuous improvement efforts in estimating risk were part of a reasonable process to assist in determining SHSP and UASI grant allocations. For example, the risk-based grant assessment model used from fiscal year 2001 through 2003 largely relied on measures of population to determine the relative risk of potential grantees, and evolved to measuring risk as the sum of threat, critical infrastructure and population density calculations in fiscal years 2004 and 2005. Further, the fiscal year 2006 process introduced a risk assessment model that included measures of Threat, Vulnerability and Consequences.

In June 2008, we reported that the way the risk-based grant assessment model measured vulnerability across states and urban areas was limited. We found that the model considered all states and urban areas equally vulnerable to a successful attack, and as a result, the final risk scores were determined exclusively by the Threat and Consequence scores. Specifically, the risk model did not measure vulnerability for each state and urban area; rather it assigned a vulnerability score of 1.0 to every state and urban area. We recommended that DHS and FEMA formulate a methodology to measure variations in vulnerability across states and urban areas. DHS components concurred with our recommendation to measure vulnerability in a way that captures variations across states and urban areas and apply this measure in future iterations of FEMA’s model. In August 2011, FEMA reported that the agency, in coordination with other DHS components, established a Vulnerability Index for the fiscal year 2011 risk-based grant assessment model to better capture the risk to states and urban areas, thereby addressing our recommendation.

Other Reviews of FEMA’s Risk Methodology

DHS and the National Research Council (NRC) have also performed reviews of FEMA’s risk assessment methodologies, providing their own conclusions and recommendations, since our 2008 review. For example, in 2010, the NRC reported that FEMA should strengthen its scientific practices, such as documentation, analyses to determine how changes to a model could affect its results, and peer review by technical experts external to DHS, in order to further develop an understanding of the uncertainties in its terrorism-related risk analyses.

Additionally, in 2016, the Homeland Security Advisory Council reported that processes by which FEMA uses to assess risk should be made more inclusive, comprehensive and effective. The Homeland Security Advisory Council recommended the following actions to strengthen this process:

FEMA should continue to send risk profiles to states and urban areas to promote timely and meaningful feedback, and enable FEMA to evaluate recommended adjustments.

Before each year’s budget submission, FEMA should discuss with congressional appropriators the current grant allocation mechanism.

We discuss FEMA’s progress in implementing these recommendations later in this report.

Various Factors Affected SHSP and UASI Grant Allocations to States and Urban Areas From Fiscal Years 2008 Through 2018

SHSP Allocations Reflect Both a State’s Relative Risk Score and the Minimum Allocation by Law

While all states and territories receive minimum SHSP program grant allocations by law, the risk-based grant assessment model also informs the grant allocation of the remaining funds to each state. However, for a majority of states each year, their SHSP grant awards are primarily based on a legal minimum amount. For example, in fiscal year 2012, 34 states, like New Mexico, were awarded $2,801,000, which included $2,745,000 based on the minimum amount by law, and $56,000 was based on its risk level. By contrast, New York was one of the high-risk states based on the risk model. For that same fiscal year (2012) New York received a total of $55,610,000, which included $2,745,000 based on the minimum amount by law, plus $52,865,000 based on its risk level.

Over the period from fiscal years 2008 through 2018, the number of low- risk states whose SHSP grant awards were primarily based on the legal minimum amount had varied from year to year, from 19 states in fiscal year 2008, to 37 states in fiscal year 2018, as shown in table 1.

In addition, from fiscal year 2008 through fiscal year 2018, there was a decrease in the percent of total SHSP funds awarded to states and territories based on FEMA’s risk model. The percent of total SHSP funding awarded to states and territories based on FEMA’s model ranged from a high of 63 percent in fiscal year 2009 (about $536 million of the $851 million of total SHSP funds), to 51 percent (about $149 million of $294 million of total SHSP funds) for fiscal year 2012. For fiscal year 2018, the total SHSP funds awarded to states and territories based on the risk-based grant assessment model was 55 percent—about $220 million of $402 million.

For specific details on SHSP grant allocations for fiscal years 2008 through 2018 by states and territories, see appendix I, table 4.

UASI Grantee Eligibility and Allocations Reflect Results from FEMA’s Risk- Based Grant Assessment Model

The UASI program uses FEMA’s risk-based grant assessment model to identify which of the 100 of the nation’s largest urban areas are eligible for grant awards in a particular fiscal year. Then, FEMA’s risk model also helps inform DHS leadership’s decisions on the final funding amounts for each grantee, according to FEMA officials. Specifically, FEMA annually assesses the risk of the 100 most populous metropolitan statistical areas—a geographical region with a relatively high population density at its core and close economic ties throughout the area—as defined by the Office of Management and Budget, in determining the eligible urban areas. From these 100 eligible urban areas, the risk-based grant assessment model identifies those urban areas that reflect recent congressional intent that up to eighty-five percent (85%) of nationwide risk is funded each year. Those urban areas below this 85 percent threshold are ineligible for UASI grant awards in that fiscal year, according to FEMA officials.

From fiscal years 2008 through 2018, the number of UASI grantees has remained relatively stable since fiscal year 2011. As figure 2 shows, the annual number of grantees has fluctuated from fiscal years 2008 through 2018, ranging from 60 to 64 grantees during fiscal years 2008, 2009 and 2010. However, since fiscal year 2011 the number of UASI grantees has averaged 31 urban areas, with a high of 39 urban areas in fiscal year 2014 and a low of 25 urban areas in fiscal year 2013. For fiscal year 2018, 32 urban areas were UASI grantees.

For additional details on UASI grant awards for fiscal years 2008 through 2018 by urban areas, see appendix I, table 5.

Because the UASI grant program is required by annual congressional guidance to fund only those urban areas that comprise up to 85 percent of risk nationally, this eligibility cut off can result in different urban areas being eligible from one year to the next. Specifically, as we demonstrated in June 2008, the variation of risk across urban areas takes on the distribution curve illustrated in figure 3.

The few urban areas with the highest relative risk score are represented along the steep part of the relative risk curve. For example, those urban areas receiving the highest awards, informed by their risk scores and ranks, are generally the same each fiscal year: New York City, Los Angeles, and Chicago, as seen in table 2.

Those urban areas that have less relative risk are represented along the flat section of the curve. There are urban areas with less risk that may not fall within the 85 percent of risk nationally during a specific year and thus would be ineligible to receive UASI funding during that year. Table 3 lists the lowest-funded urban areas for the last 5 fiscal years, based on our analysis of the funding amounts each received within each fiscal year. For example, during the period of fiscal year 2008 through fiscal year 2018 San Antonio, Texas, and Hampton Roads, Virginia only received awards in fiscal years 2008, 2009, 2014, 2017, and 2018.

In addition to changes to urban areas’ risk ranking from one year to the next, the amount that an urban area received of the total amount of UASI funds in a given year can change. FEMA has established a process for developing grant award funding options based on the results of the risk- based grant assessment model. These funding options are provided to the Secretary of Homeland Security for consideration and final approval. According to FEMA officials, the options may vary each year based on DHS leadership’s priorities and concerns at the time; however, all options represent only those eligible grantees that represent up to 85 percent of the nation’s risk, as determined by the risk-based grant assessment model. In fiscal year 2013, FEMA shifted its UASI grant funding to a process referred to as “funding bands.” In fiscal year 2018, for example, UASI grantees such as Orlando, Florida; Hampton Roads, Virginia; and San Antonio, Texas each received a $1.5 million UASI grant, whereas a grouping of UASI grantees that included Sacramento, California; Pittsburgh, Pennsylvania; and Portland, Oregon each received $2.5 million.

According to FEMA officials, grouping jurisdictions with similar risk scores into funding bands is an effort to stabilize and retain grantees’ funding levels over multiple years, as annual UASI grants will fund projects that are multiyear investments and carried out over a 24 to 36-month performance period. For example, if one jurisdiction increased by four ranks and another jurisdiction in the same group dropped six ranks, the two jurisdictions would stay in the same funding band if the overall risk scores remained close together. The purpose of the funding bands is to ensure that some consistency in funding exists for jurisdictions, given minor changes in the relative risk ranking. FEMA looks at the natural risk breaks and historical grant allocation data for each year. For example, each year FEMA presents for consideration by DHS leadership the historical funding and the number of urban areas that have been placed in specific funding bands in prior grant years, if any, and the differences between the relative risk scores in the current fiscal year. According to FEMA officials, the last few grant years had produced similar funding bands, which are subject to change depending on DHS leadership’s final decisions.

FEMA Has Improved Its Risk-based Grant Assessment Model, but Additional Steps Could Further Strengthen Its Model

FEMA Has Taken a Number of Steps to Improve the Risk-based Grant Assessment Model for Allocating SHSP and UASI Grants

Since 2008, FEMA has taken a number of steps to assess and improve its risk-based grant assessment model for allocating grants based on past reviews, our prior recommendations, and various changes related to evolving terrorist threats and real-world scenarios. For example, FEMA added a Vulnerability Index to its risk model in 2011 in response to our 2008 recommendation. Most recently, for fiscal year 2018, FEMA has included a “soft target index.” According to FEMA officials, this index was added to account for the current threat for areas where crowds congregate. Figure 4 illustrates the timeline of FEMA changes to the risk- based assessment model and prior assessments.

Figure 5 depicts the risk-based grant assessment model used for fiscal year 2018 SHSP and UASI grant awards.

Figure 6 depicts the changes in the Threat, Vulnerability, and Consequence indexes used in the risk-based grant assessments model for fiscal year 2008, compared to 2018. As we noted above, the 2008 risk model did not measure Vulnerability for each state and urban area, and risk scores were essentially determined by Threat and Consequences indexes.

Changes to the Consequence Index can have the most impact on the relative risk scores because of the weight of this index (50 percent), relative to the weights for the Threat and Vulnerability indexes. Further, the weight for population within the Consequence Index represented 30 percent of the total fiscal year 2018 risk model value. As a result, the weight for the population index was greater than the weights of either the Threat Index or Vulnerability Index, each 25 percent. FEMA has decreased the weight for the population index over time, from 40 percent in 2008 to 30 in 2011, where it has remained consistent through 2018.

For fiscal year 2018, FEMA modified how the population index was calculated within the Consequence Index to better account for attacks staged by individuals, so-called lone wolves. FEMA did so, in part, by reducing the importance of population density within the population index. In past risk models, the population index had favored high-density, high- rise urban areas, commensurate with building destruction scenarios — the 9/11-style attack scenarios that focused on large building destruction events, according to FEMA officials. The 2018 change to cap population density in the population index reduces the impact those extremely-dense population areas have in the methodology, according to FEMA officials.

The other measures used to make up the Consequence Index remain relatively unchanged since our review in 2008, although FEMA has renamed the indexes.

Vulnerability Index

As explained earlier, FEMA added a Vulnerability Index to its risk-based grant assessment model in 2011, in response to our 2008 recommendation. According to FEMA officials, the Vulnerability Index helps support what DHS is trying to protect, primarily the protection of citizens and critical infrastructure. For example, the Vulnerability Index includes a measure designed to assess the extent that certain types of national critical infrastructure assets may be considered for possible attack. This Targeted Infrastructure Index measure uses actionable intelligence on types of critical infrastructure targets, such as aviation, mass transit and commuter rail. FEMA works with DHS’s National Protection and Programs Directorate to match its critical infrastructure dataset to actionable intelligence from DHS’s Office of Intelligence & Analysis to compile this measure.

Vulnerability Index Designed to measure the likelihood of a successful attack in a state or urban area, based on a) intelligence information of those critical infrastructure assets identified by foreign or domestic terrorists; b) the extent of international borders entries (land, sea and air) located in a state or urban area, and c) special events where crowds congregate and are susceptible to homegrown extremism and lone wolf attacks.

For the fiscal year 2018 grant, FEMA has included a “soft target index.” According to FEMA officials, this index was added to account for the current threat for areas where crowds congregate. Based on previous feedback received through this process, FEMA updated the fiscal year 2018 risk methodology to better account for the nation’s current threat environment. The soft target index is composed of two new data elements:

Visitors—domestic and international—using the same data used in the calculation of the Population Index; and

Special events measure—uses Special Event Assessment Rating data from DHS Office of Operations Coordination to identify large events that are state and local events that may require federal assistance. Examples of such events include the Super Bowl, the Boston Marathon and New Year’s Eve in Times Square.

In fiscal year 2018, FEMA added a new “isolation” measure to account for the challenges of response for those states, territories, and urban areas outside the contiguous United States, who rely on prompt mutual aid from neighboring jurisdictions. According to FEMA officials, the isolation data element was included as a response to challenges the agency witnessed as a result of the 2017 Hurricane season, specifically the unique challenges of distant U.S. territories receiving timely mutual aid from other states. For example, if Hawaii, Guam or American Samoa were attacked, there would be little to no outside help for a number of days. As a result, FEMA modified the fiscal year 2018 Border Crossings data element weight, which was dropped from 6 percent to 4 percent, in order to establish a 2 percent weight for the isolation measure.

Threat Index

The weight of the Threat Index was raised from 20 percent to 30 percent in fiscal year 2011, and has been modified again for fiscal year 2018. Specifically, according to FEMA and DHS officials, DHS leadership made a policy decision to reduce the Threat Index’s weight from 30 percent in 2017, to 25 percent in 2018, due to the change in current threat environment, since Congress directed FEMA in the Explanatory Statement accompanying the FY 2017 DHS Appropriations Act to review the risk model to account for this changing threat environment. FEMA officials further stated that they assumed, as domestic terrorism and soft targets are considered to be prevalent nationwide and pose more of a challenge in identifying the source of actionable threats. FEMA officials stated that this modification to the Threat Index better reflects real-world scenarios.

Since fiscal year 2012, FEMA has included information on domestic terrorism as well as international terrorism in its Threat Index. According to DHS officials, home grown extremism is also a likely threat, often through lone wolf attacks. DHS officials decided to assign all urban areas a minimum threat score to reflect the fact that all areas have some level of threat. According to DHS officials, the addition of a domestic terror threat measure resulted in a decrease in the variation of threat scores across states and urban areas. According to DHS officials, lone wolf attacks are difficult to determine who the actors may be, or when and where they will attack.

Stakeholder Feedback

FEMA annually transmits risk profile information to states and urban areas to promote timely and meaningful feedback. According to FEMA officials, draft risk profiles are sent to all 56 states and territories and 100 eligible urban areas closely after the enactment of DHS’s annual appropriations. States and urban areas are given a 2-week period prior to the release of the Notices of Funding Opportunity to review their draft risk profiles and provide FEMA any comments or data corrections that should be considered. According to FEMA officials, it encourages and welcomes stakeholders to make suggestions for new or different data sets for the subsequent fiscal year's risk assessment at any time during the year convenient to the stakeholder. FEMA also conducts webinars during this period to can explain the risk profiles in detail, as well as discuss any updates to data sets and/or any enhancements to the risk assessment. This will often result in feedback on data elements and the methodology of the risk-based grant assessment model, according to FEMA officials. According to FEMA officials, this feedback process has been used to help guide FEMA’s consideration of enhancements to the risk-based grant assessment model. For example, FEMA officials noted that this process helped them in their efforts to develop the soft targets index into the 2018 risk model.

FEMA Does Not Fully Make Use of Recognized Scientific Practices in Maintaining Its Risk Assessment Model

In 2010, the National Research Council (NRC) recommended that incorporating scientific practices can provide decision makers a further understanding of the effects of its policy judgments and assumptions—i.e. addressing uncertainties—in its terrorism-related risk analyses. The NRC identified “good scientific practice” for model-based work. Specifically, the NRC recommended that detailed documentation for all risk models, including rigorous mathematical formulations, be implemented department-wide. Additionally, the NRC recommended that all risk models undergo verification and validation—or a sensitivity analysis at the least—of its risk-based grant assessment model. Finally, the NRC recommended that FEMA should undertake an external peer review by technical experts outside of DHS, and review its risk-informed formulas in order to identify issues such as logic flaws, evaluate the ramifications of the choices of weightings and parameters, and improve the risk model’s transparency. However, FEMA has not fully adopted these scientific practices for its risk-based grant assessment model.

Documentation: FEMA documentation on the sources of data used for the model’s calculations does not include information that would enable a reviewer to understand the underlying assumptions that form the basis for its risk-based grant assessment model—such as the size of the weights assigned to Threat, Vulnerability, and Consequence, or the justification for changes to these weights from one year to the next. FEMA officials stated that they focus their limited time and resources on developing the executive summary-level materials that DHS leadership will use to determine final grant eligibility and grant allocation amounts. Also, to a lesser extent, FEMA officials said they rely on the expertise of the subject matter experts from DHS’s Office of Intelligence and Analysis, and DHS’s National Protection and Preparedness Division’s Office of Cyber and Infrastructure Analysis, parts of DHS that contribute to the annual risk assessment process.

In April 2018, we identified documentation as one of the key methodological elements to the baseline structure of an economic analysis. Specifically, the elements include that the analysis is clearly written with a plain language summary, has clearly labeled tables that describe the data used and results, and has a conclusion that is consistent with these results. The analysis cites all sources used and documents that it is based on the best available economic information. The analysis documents that it complies with a robust quality assurance process and, where applicable, the Information Quality Act, and should disclose the use and contributions of contractors and outside consultants. FEMA officials agreed with our analysis of FEMA’s supporting documentation, and officials stated that maintaining additional documentation could further assist reviewers. Documenting how subject matter expert assumptions are made would help FEMA increase the transparency of the model for key internal and external stakeholders.

In-Depth Analyses: Similarly, we could not determine whether FEMA sufficiently performed all the analyses of the model’s sensitivity needed to determine how changes to its risk-based grant assessment model could affect the resulting risk scores. FEMA officials stated that they have only analyzed the effect of a data element when it has been added to the model (e.g.: the Soft Target Index in 2018). Further, FEMA officials were unable to provide us with documentation on their sensitivity analyses processes or their results.

DHS’s Risk Lexicon states that sensitivity analysis can be used to examine how individual variables can affect the outputs of risk assessment methodologies. In addition, OMB Circular A-94 recommends that the outcomes from a risk model should be analyzed to determine how sensitive such outcomes are to changes in the model’s assumptions. The assumptions that deserve the most attention will depend on the dominant elements and the areas of greatest uncertainty of the program being analyzed. In addition, research in the actuarial sciences also states that sensitivity analysis “is of fundamental importance to risk analysts, especially in the presence of complex computational models with uncertain inputs.”

As we stated earlier, understanding the extent that uncertainty has on the results of the model can be especially important if the model produces materially different results in response to even small changes in assumptions—often referred to as the “sensitivity” or “robustness” of a model’s assumptions and results. We have reported on FEMA’s risk- based grant assessment model in June 2008 and March 2013, where we found grant years when the risk model was sensitive to even small changes. For example, we noted that a potential increase or decrease in a measure would have resulted in one urban area displacing the eligibility of another, thereby potentially shifting funding as well. FEMA officials stated that they focus their limited time and resources on developing the executive summary-level materials that DHS leadership will use to determine final grant eligibility and grant allocation amounts. FEMA officials agreed that they could better document the steps used in their analyses across all the model’s measures and weights so that a complete understanding of potential impacts are documented and can be made available to leadership when making decisions about changes.

FEMA’s implementation of sensitivity analyses could help the agency to assess changes to the risk-based grant assessment model including the introduction of new data elements into Threat, Vulnerability, and Consequence indexes, the modifications to how existing data elements are calculated, and the changing of the weights assigned to the Threat, Vulnerability, and Consequence indexes. Further, FEMA’s implementation of sensitivity analyses has the ability to show decision makers the impact or predicted impact of adjustments to FEMA’s risk- based grant assessment model, including with potential shifts in funding towards or away from certain grantees.

Use of External Peer Review: FEMA has not subjected its risk-based grant assessment model to a peer review by independent, external technical experts, as previously recommended in 2010 by the NRC. According to FEMA officials, its risk assessment methodology has undergone comprehensive internal reconsideration over time to better reflect real-world scenarios, but such reviews have not included external peer reviews. FEMA officials stated that its risk-based grant assessment model has gone through past reviews including a review as part of DHS’s quadrennial review in 2014, and the model is reviewed by internal subject matter experts from DHS’s Office of Intelligence and Analysis, and DHS’s National Protection and Preparedness Division’s Office of Cyber and Infrastructure Analysis as part of the annual risk assessment process. FEMA officials stated that the agency is exploring the possibility of participating in a DHS collaborative group to internally review and provide feedback on the model’s underlying assumptions and methods. Such a group could review the underlying components of the current risk-based grant assessment model and suggest improvements, as well as present and evaluate other risk assessment theories and approaches. FEMA officials told us they have encountered time and resources constraints on establishing an external peer review process.

As we have previously reported, independent external peer reviews can increase the probability of success by improving the technical quality of projects and the credibility of the decision-making process, and provide reasonable assurance that the agency’s approach is reproducible and defensible. In addition, in December 2004, OMB issued the memorandum “Final Information Quality Bulletin for Peer Review” which established government-wide guidance aimed at enhancing the practice of peer review of government science documents. OMB noted that peer review can increase the quality and credibility of the scientific information generated across the federal government, which was an effort to improve the quality of the scientific information upon which policy decisions are based. OMB also noted that, while peer review may take a variety of forms, agencies will need to consider at least the following issues when coordinating an external peer review: individual versus panel review; timing; scope of the review; selection of reviewers; disclosure and attribution; public participation; disposition of reviewer comments; and adequacy of prior peer review.

These scientific processes are designed to help decision makers better understand the impact or predicted impact of risk management alternatives, and provide greater confidence in the reliability of the risk assessment model’s results. Full implementation of these processes better position FEMA to provide further assurances that their risk-based grant assessment model and grant allocation approaches are reasonable, of high-quality, and credible.

Conclusions

Given that risk management has been endorsed by the federal government as a way to direct finite resources to states and those urban areas that are most at risk of terrorist attack, it is important that FEMA’s risk-based grant assessment model supports the application of policy judgments and analytic assumptions in the model’s role of allocating those limited resources. Decreased funding levels for SHSP and UASI grant programs have increased the importance of using risk management techniques to more effectively target finite federal dollars. DHS and FEMA have strengthened its risk-based grant assessment model for allocating grants, taking into account analysis and recommendations from a variety of reviews. These improvements include the addition of a Vulnerability Index and modifications to the Threat Index. We have identified opportunities where FEMA could strengthen its scientific practices. First, documenting the model’s underlying assumptions and the results of sensitivity analysis can assist decision makers in better understanding the predicted impact of risk management alternatives. Second, expanding the use of sensitivity analysis could further enhance the model. Developing a greater understanding of the how uncertainty affects its risk-based grant assessment model’s results helps achieve the objectives of risk management. Third, coordinating an independent external peer review of the methodology of its risk-based grant assessment model would better position the agency to provide reasonable assurance that FEMA’s risk model and grant allocation approach that FEMA uses for its SHSP and UASI programs are reasonable, of high-quality, and credible. Applying such scientific practices could assist FEMA in further strengthening its risk-based grant assessment model.

Recommendations for Executive Action

We are making the following three recommendations to FEMA.

The FEMA Administrator should fully document the underlying assumptions and justifications that form the basis of the risk-based grant assessment model, such as the size of the weights assigned to Threat, Vulnerability, and Consequence, or the justification for changes to these weights from one year to the next.

The FEMA Administrator should perform sensitivity analyses to verify how changes to the risk-based grant assessment model could affect the resulting risk scores, and document the results.

The FEMA Administrator should take steps to coordinate an independent, external peer review of its risk-based grant assessment model.

Agency Comments and Our Evaluation

We provided a draft of this product to the FEMA and DHS for comment. In its comments, reproduced in appendix II, FEMA generally concurred with our findings and three recommendations.

In FEMA’s concurrence to our first recommendation that the agency fully document the underlying assumptions and justifications that form the basis of the risk-based grant assessment model, FEMA requested that GAO consider this recommendation resolved and closed as implemented. As part of FEMA’s response, they reiterate their process of providing draft Risk Profiles to all 100 urban areas and 56 states and territories and their annual communications to Congress on how FEMA calculated risk and computed grant awards. We recognized FEMA’s stakeholder feedback efforts in this report. However, as we noted, FEMA’s documentation on the sources of data used for the model’s calculations does not include information that would enable a reviewer to understand the underlying assumptions that form the basis for its risk-based grant assessment model. Further, as stated earlier, documentation is one of the key methodological elements to the baseline structure of this type of analysis, documenting that it complies with a robust quality assurance process and, where applicable, the Information Quality Act, and should disclose the use and contributions of contractors and outside consultants. In order to fully implement this recommendation, documenting how subject matter expert assumptions are made would help FEMA increase the transparency of the model for key internal and external stakeholders, and will further support the efforts of an independent external peer review of FEMA’s risk-based assessment model.

Regarding the second recommendation, FEMA concurred, stating that the agency will expand the use of sensitivity analysis to review the entire risk methodology, and will also document these results for leadership review, as appropriate. Finally, regarding the third recommendation, FEMA concurred, stating that they will coordinate an independent external peer review and develop a detailed written response to leadership for further appropriate action.

FEMA and DHS also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, and other interested parties. This report will also be available at no charge on our Web site at http://www.gao.gov. Should you or your staff have any questions concerning this report, please contact me at (202) 512-8777 or CurrieC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report.

Appendix I: Grant Funding and Awards for State Homeland Security Grant Program (SHSP), and the Urban Area Security Initiative (UASI) Grant Program for Fiscal Years 2008 Through 2018

Fiscal Years Mississippi Total Award Amount above legal minimum

Fiscal Years West Virginia Total Award Amount above legal minimum

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Chris P. Currie, at (202) 512-8777 or CurrieC@gao.gov.

Staff Acknowledgments

In addition, key contributors to this report were Aditi Archer, Chris Keisling, Assistant Director, John Vocino, Analyst-in-Charge, Chuck Bausell, Dominick Dale, Dorian Dunbar, Eric Hauswirth, Serena Lo, Heidi Nielson, and Hadley Nobles.