Homeland Security: DHS Risk-Based Grant Methodology Is Reasonable, But Current Version's Measure of Vulnerability is Limited
Released: 2008-06-27T00:00:00

Letter

Background

Risk management has been endorsed by Congress, the President, and the Secretary of DHS as a way to direct finite resources to those areas that are most at risk of terrorist attack under conditions of uncertainty. The purpose of risk management is not to eliminate all risks, as that is an impossible task. Rather, given limited resources, risk management is a structured means of making informed trade-offs and choices about how to use available resources effectively and monitoring the effect of those choices. Thus, risk management is a continuous process that includes the assessment of threats, vulnerabilities, and consequences to determine what actions should be taken to reduce or eliminate one or more of these elements of risk.

To provide a basis for examining efforts at carrying out risk management, GAO developed a framework for risk management based on best practices and other criteria. The framework is divided into five phases: (1) setting strategic goals and objectives, and determining constraints; (2) assessing the risks; (3) evaluating alternatives for addressing these risks; (4) selecting the appropriate alternatives; and (5) implementing the alternatives and monitoring the progress made and the results achieved (see Fig.1).

Because we have imperfect information for assessing risks, there is a degree of uncertainty in the information used for risk assessments (e.g., what the threats are and how likely they are to be realized). As a result, it is inevitable that assumptions and policy judgments must be used in risk analysis and management. It is important that key decision-makers understand the basis for those assumptions and policy judgments and their effect on the results of the risk analysis and the resource decisions based on that analysis.

DHS has used an evolving risk-based methodology to identify the urban areas eligible for HSGP grants and the amount of funds states and urban areas receive (see Fig 2). For example, the risk analysis model used from fiscal year 2001 through 2003 largely relied on measures of population to determine the relative risk of potential grant recipients, and evolved to measuring risk as the sum of threat, critical infrastructure and population density calculations in fiscal years 2004 and 2005.

The fiscal year 2006 process introduced assessments of threat, vulnerability and consequences of a terrorist attack in assessing risk. In addition to modifications to its risk analysis model, DHS adopted an effectiveness assessment for fiscal year 2006 to determine the anticipated effectiveness of the various risk mitigation investments proposed by urban areas, which affected the final amount of funds awarded to eligible areas. For the fiscal year 2007 allocation process, DHS defined Risk as the product of Threat times Vulnerability and Consequences, or “R= T* (V & C)” and applied a three-step risk-based allocation methodology which incorporates analyses of risk and effectiveness to select eligible urban areas and allocate UASI and SHSP funds (see Fig. 3). The three steps include: 1. Implementation of a Risk Analysis model to calculate scores for states and urban areas, defining relative Risk as the product of Threat, Vulnerability and Consequences; 2. Implementation of an Effectiveness Assessment, including a process where state and urban area representatives acting as peer reviewers assess and score the effectiveness of the proposed investments submitted by the eligible applicants. This process is also known as peer review. 3. Calculation of a Final Allocation of funds based on states’ and urban areas’ risk scores as adjusted by their effectiveness scores.

The Post-Katrina Emergency Management Reform Act places responsibility for allocating and managing DHS grants with the Federal Emergency Management Agency (FEMA). While FEMA is responsible for implementing the above 3-step process, FEMA relies on other DHS components such as the National Protection and Programs Directorate (NPPD) and the Office of Intelligence and Analysis (I&A) in the development of the risk analysis model, which we will discuss in greater detail below.

Risk Analysis Model

DHS employs a risk analysis model to assign relative risk scores to all states and urban areas under the SHSP and UASI grant programs. These relative risk scores are also used to differentiate which urban areas are eligible for UASI funding. These eligible areas are divided into two tiers: Tier 1 UASI grantees and those eligible for Tier 2. In fiscal year 2007, 45 candidates were eligible to apply for funding under the UASI program, and eligible candidates were grouped into two tiers according to relative risk. Tier 1 included the six highest risk areas; Tier 2 included the other 39 candidate areas. Figure 4 provides an overview of the factors that are included in the risk analysis model for fiscal year 2007 and their relative weights. The maximum relative risk score possible for a given area was 100. The Threat Index accounted for 20 percent of the total risk score; the Vulnerability and Consequences Index accounted for 80 percent.

The Threat Index accounted for 20 percent of the total risk score, which was calculated by assessing threat information for multiple years (generally, from September 11, 2001 forward) for all candidate urban areas and categorizing urban areas into different threat tiers. According to DHS officials, the agency’s Office of Intelligence and Analysis (I&A) calculated the Threat Index by (1) collecting qualitative threat information with a nexus to international terrorism, (2) analyzing the threat information to create threat assessments for states and urban areas, (3) empanelling intelligence experts to review the threat assessments and reach consensus as to the number of threat tiers, and (4) assigning threat scores. This process, according to DHS officials, relied upon analytical judgment and interaction with the Intelligence Community, as opposed to the use of total counts of threats and suspicious incidents to calculate the Threat Index for the 2006 grant cycle. The final threat assessments are approved by the Intelligence Community—the Federal Bureau of Investigation, Central Intelligence Agency, National Counterterrorism Center, and the Defense Intelligence Agency—along with the DHS Under Secretary for Intelligence and Analysis and the Secretary of DHS, according to DHS officials.

The Vulnerability and Consequences index accounts for 80 percent of the total risk score. Because DHS considered most areas of the country equally vulnerable to a terrorist attack given freedom of movement within the nation, DHS assigns vulnerability a constant value of 1.0 in the formula across all states and urban areas. Therefore, DHS’s measurement of vulnerability and consequences is mainly a function of the seriousness of the consequences of a successful terrorist attack, represented by four indices: a Population Index, an Economic Index, a National Infrastructure Index, and a National Security Index.

Population Index (40 percent). This index included nighttime population and military dependent populations for states and urban areas, based upon U.S. Census Bureau and Department of Defense data. For urban areas, factors such as population density, estimated number of daily commuters, and estimated annual visitors were also included in this variable using data from private entities. DHS calculated the Population Index for urban areas by identifying areas with a population greater than 100,000 persons and cities that reported threat data during the past year, then combined cities or adjacent urban counties with shared boundaries to form single jurisdictions, and drew a 10-mile buffer zone around identified areas.

Economic Index (20 percent). This index is comprised of the economic value of the goods and services produced in either a state or an urban area. For states, this index was calculated using U.S. Department of Commerce data on their percentage contribution to Gross Domestic Product. For UASI urban areas, a parallel calculation of Gross Metropolitan Product was incorporated.

National Infrastructure Index (15 percent). This index focused on over 2,000 critical infrastructure/key resource (CIKR) assets that were identified by DHS’s Office of Infrastructure Protection. These particular critical infrastructure assets are divided into two rankings that, if destroyed or disrupted, could cause significant casualties, major economic losses, or widespread/long term disruptions to national well-being and governance capacity. The Tier 2 CIKR assets include the nationally-significant and high-consequence assets and systems across 17 sectors. Tier 1 assets are a small subset of the Tier 2 list that include assets and systems certain to produce at least two of four possible consequences if disrupted or destroyed: (1) prompt fatalities greater that 5,000; (2) first-year economic impact of at least $75 billion; (3) mass evacuations with prolonged (6 months or more) absence; and (4) loss of governance or mission execution disrupting multiple regions or critical infrastructure sectors for more than a week, resulting in a loss of necessary services to the public. Tier 1 assets were weighted using an average value three times as great as Tier 2 assets.

The National Security Index (5 percent). This index considered three key national security factors: whether military bases are present in the state or urban area; how many critical defense industrial base facilities are located in the state or urban area; and the total number of people traversing international borders. Information on these inputs comes from the Department of Defense and DHS.

Effectiveness Assessment

In addition to determining relative risk using the risk analysis model, DHS added an effectiveness assessment process in fiscal year 2006 to assess and score the effectiveness of the proposed investments submitted by grant applicants. To assess the anticipated effectiveness of the various risk mitigation investments that states and urban areas proposed, DHS required states and urban areas to submit investment justifications as part of their grant applications. The investment justifications included up to 15 “investments” or proposed solutions to address homeland security needs, which were identified by the states and urban areas through their strategic planning process. DHS used state and urban area representatives as peer reviewers to assess these investment justifications. The criteria reviewers used to score the investment justifications included the following categories: relevance to national, state and local plans and policies such as the National Preparedness Guidance states’ and urban areas’ homeland security plans, anticipated impact, sustainability, regionalism, and the applicants’ planned implementation of each proposed investment. Reviewers on each panel assigned scores for these investment justifications, which, according to DHS officials, were averaged to determine a final effectiveness score for each state and urban area applicant.

In fiscal year 2007, DHS provided states and urban areas the opportunity to propose investment justifications that included regional collaboration to support the achievement of outcomes that could not be accomplished if a state or urban area tried to address them independently. States and urban areas could choose to submit multi-state or multi-urban area investment justifications which outlined shared investments between two or more states or between two or more urban areas. Such investments were eligible for up to 5 additional points on their final effectiveness score, or up to 8 more effectiveness points for additional proposed investments, although these additional points would not enable a state’s or urban area’s total effectiveness score to exceed 100 points. These proposed investments were reviewed by one of two panels established specifically to consider multi-applicant proposals. Points were awarded based on the degree to which multi-applicant investments showed collaboration with partners and demonstrated value or outcomes from the joint proposal that could not be realized by a single state or urban area.

Final Allocation Process

DHS allocated funds based on the risk scores of states and urban areas, as adjusted by their effectiveness scores. DHS officials explained that while allocations are based first upon area risk scores, the effectiveness scores are then used to determine adjustments to states and urban areas allocations based on an “effectiveness multiplier.” States and urban areas with high effectiveness scores received an additional percentage of their risk-based allocations, while states and urban areas with low effectiveness scores had their risk-based allocations lowered by a percentage.

In addition to determining funding by risk score as adjusted by an effectiveness multiplier, urban areas that received funds through the UASI grant program were subject to an additional tiering process that affected funding allocation. For example, in fiscal year 2007, the 45 eligible urban area candidates were grouped into two tiers according to relative risk. The Tier 2 UASI grantees included the 6 highest-risk areas; Tier 2 UASI grantees included another 39 candidate areas ranked by risk. The 6 Tier 1 UASI grantees were allocated fifty-five percent of the available funds, or approximately $410.8 million, while the 39 Tier 2 UASI grantees received the remaining forty-five percent of available funds, or approximately $336.1 million.

Shifting to Urban Area Boundaries Defined by MSA was the Primary Change to DHS’s Risk-Based Methodology in 2008

DHS’s risk-based methodology had few changes from fiscal year 2007 to 2008. DHS changed the definition it used to identify the UASI areas included in the risk analysis model in 2008 from an urban area’s center city plus a ten-mile radius to metropolitan statistical areas (MSAs) as defined by the Census Bureau. DHS made this change in response to the 9/11Act requirement to perform a risk assessment for the 100 largest MSAs by population. Because the change in definition generally expanded the geographic area of each potential UASI grant recipient, the change had an effect on the data used to assess threat and consequences, and it may also have resulted in the use of more accurate data in the risk analysis model. The change to the use of MSA data in fiscal year 2008 also resulted in changes in the relative risk rankings of some urban areas. As a result, DHS officials expanded the eligible urban areas in fiscal year 2008 to a total of 60 UASI grantees, in part, to address the effects of this change to MSA data, as well as to ensure that all urban areas that received fiscal year 2007 funding also received funding for fiscal year 2008, according to DHS officials.

Changing the boundaries had an effect on the data by which risk is calculated because the change in boundaries resulted in changes in the population and critical assets within the new boundaries. Figure 3 below uses the Chicago, IL urban area to illustrate this change. One benefit of the change to MSAs was that the UASI boundaries align more closely with the boundaries used to collect some of the economic and population data used in the model. Consequently, the fiscal year 2008 model may have resulted in more accurate data. Because the 2007 boundaries were based on distance, areas inside the boundaries may have included partial census tracts or partial counties, each of which would have required DHS to develop rules as to how to handle the partial areas. By contrast, the MSAs are based on counties and allow DHS to use standard census data instead of developing an estimated population within the defined boundaries. Additional information describing the boundaries of UASI urban areas for fiscal year 2007 versus fiscal year 2008 is presented in Appendix II.

DHS calculated the Population Index of MSAs by: (1) using census data to determine the population and population density of each census tract; (2) calculating a Population Index for each individual census tract by multiplying the census tract’s population and population density figures; and (3) adding together the population indices of all of the census tracts making up the MSA. DHS did not use average population density because using an average resulted in losing information about how the population is actually distributed among the tracts. Using averages for population density over census tracts with dissimilar densities could have yielded very misleading results, according to DHS officials.

The change to MSAs for fiscal year 2008 resulted in an increase of almost 162,000 square miles across the total area of urban area footprints. While 3 urban areas actually lost square mileage because of the change, the other areas all increased their square mileage footprint by almost 2,700 square miles on average. The increased size of urban areas’ footprints increased the number of critical infrastructure assets that were counted within them. We analyzed the number of Tier 1 and Tier 2 critical infrastructure assets associated with UASI areas between fiscal year 2007 and 2008, and found a higher number of total Tier 1 and Tier 2 critical infrastructure assets assigned to urban areas in 2008, and–individually—almost all urban areas increased the number of assets assigned to them.

This change to the use of MSAs also resulted in changes in urban areas rankings, including the increase of the relative risk scores for such urban areas as Albany, Syracuse and Rochester, NY, and Bridgeport, CT. As a result, DHS officials expanded the eligible urban areas in fiscal year 2008 to a total of 60 with the top seven highest risk areas comprising UASI Tier 1 grantees, and the 53 other risk-ranked UASI Tier 2 grantees. As in fiscal year 2007, the top seven UASI Tier 1 grantee areas will receive fifty-five percent of the available funds, or approximately $429.9 million, and the remaining 53 UASI Tier 2 grantees will receive forty-five percent of the available funds, or approximately $351.7 million. According to DHS officials, the decision to expand the eligible urban areas to a total of sixty was a policy decision largely driven by two factors: the 9/11 Act requirement that FEMA use MSAs; and the desire to continue to fund urban areas already receiving funding.

DHS’s Risk-based Methodology is Generally Reasonable, But the Vulnerability Element of the Risk Analysis Model Has Limitations that Reduce Its Value

The risk-based methodology DHS uses to allocate HSGP grant dollars is generally reasonable. It includes and considers the elements of risk assessment—Threat, Vulnerability, and Consequences—and, as DHS’s risk-based methodology has evolved, its results have become less sensitive to changes in the key assumptions and weights used in the risk analysis model. Furthermore, the indices that DHS uses to calculate the variable constituting the greatest portion of the risk analysis model— Consequences—are reasonable. However, limitations such as the absence of a method for measuring variations in vulnerability reduce the vulnerability element’s value. Although DHS recognized and described the significance of Vulnerability in its FY 2006 model, the model DHS used for fiscal years 2007 and 2008 used a constant value of 1.0 in its formula, rather than measuring variations in vulnerability across states and urban areas.

DHS’s Risk Analysis Model is Reasonable Because it Contains the Key Elements of Risk Assessment, Relies on Reasonable Indices to Measure Consequences, and is Less Sensitive to Changes in Variables

One measure of the reasonability of DHS’s risk-based methodology is the extent to which DHS’s risk analysis model provides a consistent method to assess risk. Risk assessment helps decision makers identify and evaluate potential risks facing key assets or missions so that countermeasures can be designed and implemented to prevent or mitigate the effects of the risks. In a risk management framework, risk assessment is a function of Threat, Vulnerability, and Consequences, and the product of these elements is used to develop scenarios and help inform actions that are best suited to prevent an attack or mitigate vulnerabilities to a terrorist attack. Threat is the probability that a specific type of attack will be initiated against a particular target/class of targets, and analysis of threat- related data is a critical part of risk assessment. The Vulnerability of an asset is the probability that a particular attempted attack will succeed against a particular target or class of targets. It is usually measured against some set of standards, such as availability/predictability, accessibility, countermeasures in place, and target hardness (the material construction characteristics of the asset). The Consequences of a terrorist attack measures the adverse effects of a successful attack and may include many forms, such as the loss of human lives, economic costs, and adverse impact on national security. The risk analysis model used by DHS is reasonable because it attempts to capture data on threats, vulnerabilities, and consequences—the three types of information used in evaluating risk.

Because DHS considered most areas of the country equally vulnerable to a terrorist attack given freedom of movement within the nation, DHS assigns vulnerability a constant value of 1.0 in the formula across all states and urban areas. Therefore, DHS’s measurement of vulnerability and consequences is mainly a function of the seriousness of the consequences of a successful terrorist attack. Because the risk analysis model is consequences-driven, another measure of the model’s overall reasonableness is the extent to which the indices used to calculate the consequences component of the model are reasonable. As previously described, the consequences component of the model is comprised of four indices – a Population Index, an Economic Index, a National Infrastructure Index, and a National Security Index – each assigned a different weight. These indices are generally reasonable.

Both the population and economic indices are calculated from data derived from reliable sources that are also publicly available, providing additional transparency for the model. For example, according to DHS officials, the fiscal year 2008 analysis used Gross Metropolitan Product (GMP) estimates prepared by the consulting firm Global Insight for the United States Conference of Mayors and the Council for the New American City that were published in January 2007, and reported on the GMP for 2005. In addition, the National Infrastructure Index focused on over 2,000 Tier 1and Tier 2 critical infrastructure/key resource assets identified by DHS’s Office of Infrastructure Protection (IP). For both fiscal years 2007 and 2008, DHS used a collaborative, multi-step process to create the Tier 2 CIKR list. First, IP works with sector-specific agencies to develop criteria used to determine which assets should be included in the asset lists. Second, these criteria are vetted with the private-sector through sector-specific councils, who review the criteria and provide feedback to IP. Third, IP finalizes the criteria and provides it to the sector-specific agencies and State and Territorial Homeland Security Advisors (HSAs). Fourth, IP asks states to nominate assets within their jurisdiction that match the criteria. Fifth, assets nominated by states are reviewed by both the sector-specific agencies and IP to decide which assets should comprise the final Tier 2 list. For example, to identify the nation’s critical energy assets, IP will work with the Department of Energy to determine which assets and systems in the energy sector would generate the most serious economic consequences to the Nation should they be destroyed or disrupted. Further, in the fiscal year 2008 process, IP added a new, additional step to allow for the resubmission of assets for reconsideration if they are not initially selected for the Tier 2 list. In addition, the National Security Index comprises only a small fraction of the model – 5 percent – and has also evolved to include more precision, such as counting the number of military personnel instead of simply the presence or absence of military bases. To identify the nation’s critical defense industrial bases, the Department of Defense analyzes the impact on current warfighting capabilities, recovery and reconstitution, threat, vulnerability, and consequences of possible facility disruption and destruction, and other aspects.

DHS’s approach to calculating threat, which accounts for the remaining 20 percent of the model, also represents a measure of the model’s overall reasonableness. DHS uses analytical judgments to categorize urban areas’ threat, which ultimately determines the relative threat for each state and urban area. DHS has used written criteria to guide these judgments, and DHS provided us with the criteria used in both of these years for our review. The criteria are focused on threats from international terrorism derived from data on credible plots, planning, and threats from international terrorist networks, their affiliates, and those inspired by such networks. The criteria provided guidance for categorizing areas based on varying levels of both the credibility and the volume of threat reporting, as well as the potential targets of threats. Results of this process are shared with the DHS Undersecretary for Intelligence and Analysis, the FBI, and the National Counterterrorism Center, all of whom are afforded the opportunity to provide feedback on the placements. Additionally, DHS develops written threat assessments that indicate whether states are “high,” “medium,” or “low” threat states. States can provide threat information that they have collected to DHS, but in order for that information to affect a state’s tier placement and threat level, the information must be relevant to international terrorism, according to DHS officials. We reviewed several examples of these assessments from 2007, which included key findings describing both identified and potential threats to the state. The classified assessments addressed potential terrorist threats to critical infrastructure in each of the 56 states and territories. However, DHS shared assessments only with state officials who had appropriate security clearances. According to DHS officials, states without officials with sufficient clearances will receive an unclassified version of their state’s assessment for the fiscal year 2009 grant process. DHS is also developing a process by which they can share the threat assessments with UASI areas, including those UASI areas whose boundaries cross state lines; however, currently the assessments are transmitted only to the DHS state representatives and state officials, and the states and representatives are responsible for sharing the information with the UASI areas, according to DHS officials.

Another measure of the overall reasonableness of DHS’s risk analysis model is the extent to which the model’s results change when the assumptions and values built into the model, such as weights of variables, change. A model is sensitive when a model produces materially different results in response to small changes in its assumptions. Ideally, a model that accurately and comprehensively assesses risk would not be sensitive, and such a model exhibiting little sensitivity could be said to be more robust than a model with more sensitivity to changes in assumptions underlying the model. A robust calculation or estimation model provides its users greater confidence in the reliability of its results. For both fiscal years 2007 and 2008, substantial changes had to be made to the weights of any of the indices used in the risk model to calculate state and urban area risk scores before there was any movement in or out of the top 7 (or Tier 1) ranked UASI areas. In other words, the model provides DHS with a level of assurance that the highest at-risk areas have been appropriately identified. While Tier 1 UASI areas were similarly robust in both FY 2007 and FY 2008, the sensitivity of Tier 2 UASI areas to changes in the weights of indices used to calculate risk scores was significant in FY 2007, but improved in FY 2008. In FY 2007, very small changes in the weights for the indices used to quantify risk (for Tier 2 UASI areas at the eligibility cut point) resulted in changes in eligibility; however, FY 2008 results are more robust, as eligibility of urban areas is much less sensitive to changes in the index weights in the FY2008 model than it was in the FY2007 model. Appendix III provides an in-depth description of the sensitivity of the model to specific changes in the relative weights of each index for Tier 1 and Tier 2 UASI areas.

Vulnerability Element of the Risk Analysis Model Has Limitations that Reduce Its Value

Although the methodology DHS uses is reasonable, the vulnerability element of the risk analysis model—as currently calculated by DHS—has limitations that reduce its value for providing an accurate assessment of risk. DHS considered most areas of the country equally vulnerable to a terrorist attack in the risk analysis model used for fiscal years 2007 and 2008 and assigned a constant value to vulnerability, which ignores geographic differences in the social, built, and natural environments across states and urban areas. Although DHS recognized and described the significance of vulnerability in its FY 2006 model, the model used for fiscal years 2007 and 2008 did not attempt to measure vulnerability. Instead, DHS considered most areas of the country equally vulnerable to a terrorist attack due to the freedom of individuals to move within the nation. As a result, DHS did not measure vulnerability, but assigned it a constant value of 1.0 across all states and urban areas.

Last year we reported that DHS measured the vulnerability of an asset type as part of its FY2006 risk analysis. DHS used internal subject matter experts who analyzed the general attributes of an asset type against various terrorist attack scenarios by conducting site vulnerability analyses on a sample of sites from the asset type in order to catalog attributes for the generic asset. These experts evaluated vulnerability by attack scenario and asset type pairs and assigned an ordinal value to the pair based on 10 major criteria. In describing its FY 2006 methodology, DHS acknowledged that because all attack types are not necessarily applicable to all infrastructures, the values for threat must be mapped against vulnerability to represent the greatest likelihood of a successful attack. DHS also acknowledged that vulnerability of an infrastructure asset was also a function of many variables and recognized that it did not have sufficient data on all infrastructures to know what specific vulnerabilities existed for every infrastructure, what countermeasures had been deployed, and what impact on other infrastructures each asset had. At that time, DHS noted it would require substantial time and resource investment to fully develop the capability to consistently assess and compare vulnerabilities across all types of infrastructure.

Vulnerability is a crucial component of risk assessment. An asset may be highly vulnerable to one mode of attack but have a low level of vulnerability to another, depending on a variety of factors, such as countermeasures already in place. According to our risk management framework, the vulnerability of an asset is the probability that a particular attempted attack will succeed against a particular target or class of targets. It is usually measured against some set of standards, such as availability/predictability, accessibility, countermeasures in place, and target hardness (the material construction characteristics of the asset). Each of these four elements can be evaluated based on a numerical assignment corresponding to the conditional probability of a successful attack. Additionally, other research has developed methods to measure vulnerability across urban areas. For example, one study described a quantitative methodology to characterize the vulnerability of U.S. urban centers to terrorist attack for the potential allocation of national and regional funding to support homeland security preparedness and response in U.S. cities. This study found that vulnerability varied across the country, especially in urban areas. The study noted that “place matters,” and a one-size-fits all strategy ignores geographic differences in the social, built, and natural environments. Furthermore, in February of 2008 the Secretary of DHS said that “as we reduce our vulnerabilities, the vulnerabilities change as well.” However, while earlier iterations of the risk analysis model attempted to measure vulnerability, DHS’s risk analysis model now considers the states and urban areas of the country equally vulnerable to a terrorist attack and assigns a constant value to vulnerability, which ignores geographic differences.

Conclusions

In fiscal year 2008, DHS will distribute approximately $1.6 billion to states and urban areas through its Homeland Security Grant Program – a program that has already distributed approximately $20 billion over the past six years – to prevent, protect against, respond to, and recover from acts of terrorism or other catastrophic events. Given that risk management has been endorsed by the federal government as a way to direct finite resources to those areas that are most at risk of terrorist attack under conditions of uncertainty, it is important that DHS use a reasonable risk- based allocation methodology and risk analysis model as it allocates those limited resources.

DHS’s risk-based allocation methodology and risk analysis model are generally reasonable tools for measuring relative risk within a given fiscal year, considering its use of a generally-accepted risk calculation formula; key model results’ decreased sensitivity to incremental changes in the assumptions related to Tier 1 UASI grantees or the eligibility for Tier 2 UASI funding, the reliability of the consequence variable component indices, and its adoption of MSAs to calculate urban area footprints. However, the element of vulnerability in the risk analysis model could be improved to more accurately reflect risk. Vulnerability is a crucial component of risk assessment, and our work shows that DHS needs to measure vulnerability as part of its risk analysis model to capture variations in vulnerability across states and urban areas.

Recommendations

To strengthen DHS’s methodology for determining risk, we are recommending that the Secretary of DHS take the following action: Instruct FEMA, I&A, and NPPD - DHS components each responsible for aspects of the risk-based methodology used to allocate funds under the Homeland Security Grant Program - to formulate a method to measure vulnerability in a way that captures variations across states and urban areas, and apply this vulnerability measure in future iterations of this risk-based grant allocation model.

Agency Comments

We requested comments on a draft of this report from the Secretary of Homeland Security, FEMA, I&A, and NPPD, or their designees. In email comments on the draft report, FEMA and I&A concurred with our recommendation that they formulate a method to measure vulnerability in a way that captures variations across states and urban areas and apply this vulnerability measure in future iterations of the risk-based grant allocation model. FEMA, I&A, and NPPD also provided technical comments, which we incorporated as appropriate.

We are sending copies of this correspondence to the appropriate congressional committees, and the Secretary of Homeland Security.

Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. For further information about this report, please contact William Jenkins, Jr., Director, GAO Homeland Security and Justice Issues Team, at (202)-512-8777 or at jenkinswo@gao.gov. GAO staff members who were major contributors to this report are listed in appendix IV.

Appendix I: Briefing for Congressional Committees, February 11-25, 2008

For the third consecutive year, GAO has been mandated as part of DHS’s annual appropriation to review and assess the HSGP’s risk analysis model and risk-based allocation methodology for determining risk and distributing funds. We responded to the mandate in February 2008 by briefing the staffs of congressional committees on the results of this review. During the course of our engagement, we had ongoing dialog with DHS officials regarding the extent to which written criteria were used in the development of the Threat Index. At that time, officials from DHS’s Office of Intelligence and Analysis stated that the criteria were not documented. As a result, we noted in the accompanying presentation slides that DHS’s approach to measuring threat did not include specific, written criteria to use when determining the threat tiers into which states and urban areas are placed.

As part of GAO’s agency protocols, we convened an exit conference with DHS officials which occurred on April 14, 2008. We provided them with a statement of facts to reflect the information gathered during our engagement. At this exit conference an official from the Office of Intelligence and Analysis said DHS had used criteria in 2007 and 2008 for categorizing cities and states based on threat, and in further discussions with DHS we were able to independently review these documents and confirm that such criteria were used in the development of the Threat Index, which is reflected in the letter above. However, we did not modify the accompanying presentation contained in this appendix.

Homeland Security Grant Program

Introduction

According to the Department of Homeland Security (DHS), in fiscal

DHS provided approximately $1.7 billion to states and urban areas through its Homeland Security Grant Program (HSGP) to prevent, protect against, respond to, and recover from acts of terrorism or other catastrophic events. DHS plans to distribute about $1.6 billion for these grants in fiscal year 2008.

The HSGP risk-based allocation process is used for the State Homeland Security Program (SHSP) and Urban Area Security Initiative (UASI).

In addition, DHS used this same approach to allocate $655 million in fiscal year 2007 under the Infrastructure Protection Program.

Objectives

In response to a legislative mandate and discussions with

relevant congressional staff, we addressed the following questions: 1. What methodology did DHS use to allocate HSGP funds for fiscal years 2007 and 2008, including any changes DHS made to the eligibility and allocation processes for fiscal year 2008 and the placement of states and urban areas within threat tiers, and why? 2. How reasonable is DHS’s methodology?

Scope and Methodology

We analyzed DHS documents including the FY2007 and FY2008 risk analysis models,

grant guidance, presentations, and interviewed DHS officials about: The HSGP grant determination process in FY07—and any changes to the The process by which DHS’s risk analysis model is used to estimate relative risk: Risk = Threat*(Vulnerability & Consequences); How the effectiveness assessment process is conducted; How final allocation decisions are made.

DHS’s methodology for ranking grantees by tiered groups and the impact of this ranking on funding allocations.

We did our work from September 2007 and February 2008, in accordance with generally accepted government accounting standards (GAGAS).

Background:

We’ve reviewed this program for the last 3 years. In previous

DHS has adopted a process of “continuous improvement” to its methods for estimating risk and measuring applicants’ effectiveness.

Inherent uncertainty is associated with estimating risk of terrorist attack, requiring the application of policy and analytic judgments. The use of sensitivity analysis can help to gauge what effects key sources of uncertainty have on outcomes.

Sensitivity of the risk analysis

In FY 2007, DHS had developed a greater understanding of the

sensitivity of the risk model as a result of its changes to the model.

GAO’s analysis of the FY 2007 model:It takes sizable changes to the weights of these indices used to quantify risk to change the areas that compose the Tier 1 list.

For those urban areas ranked near the bottom of Tier 2 list, very small changes in the weights for the indices used to quantify risk can result in changes in eligibility.

According to DHS officials, there were a number of changes in the rankings, and these changes were driven by the required change in FY2008 to use MSAs.

Effectiveness Assessment

For fiscal year 2007 DHS assessed the applications submitted by states and eligible urban areas.

DHS used a peer-review process to assess and score the effectiveness of proposed investments by: Engaging the states in identifying and selecting peer Having peer reviewers individually score investments, and Assigning peer reviewers to panels to make final effectiveness score determinations.

FY 2007 Effectiveness Assessment

Final Allocation Process – FY 2007 Grants Based on Both Risk and Effectiveness Scores

DHS allocated funds based on the risk scores of states and urban areas, as adjusted by their effectiveness scores.

SHSP provided a minimum allocation, ensuring no state or territory’s allocation falls below the minimum levels established by the USA PATRIOT Act.

For UASI, DHS established maximum and minimum allocation to minimize variations in some urban areas’ final allocations between years. minimum = 0.375% of all funds appropriated for SHSP and UASI.

Final Allocation Process – Ranking UASI Grantees by Tiered groups

Fiscal year 2007, 45 eligible candidates were grouped into

two tiers according to relative risk.

Tiering was established from a policy judgment by DHS leadership, according to DHS grant officials.

Tier I included the 6 highest risk areas; Tier II included the other 39 candidate areas ranked by risk.

FY 2007 Tier I Urban Areas = 6 Urban Areas, $410,795,000 allocated (55 percent of available funds).

FY 2007 Tier II Urban Areas = 39 Urban Areas, $336,105,000 allocated (45 percent of available funds).

60 eligible UASI areas in FY 2008: Tier I = 7 highest risk areas and eligible for 55 percent of

available funds -- $429,896,500.

Tier II = 53 areas (14 more than FY 2007) and eligible for 45 percent of available funds -- $351,733,500.

According to DHS officials, the expansion to 60 eligible UASI areas for FY2008 is a policy decision largely driven by two factors: 1. The new requirement that FEMA use MSAs; 2. The desire to remain consistent with the funding.

Observations on the Reasonableness of the HSGP Grant Distribution Methodology

Observations on the Reasonableness of the HSGP Grant Distribution Methodology (continued)

DHS could take an additional step to evaluate the reliability and validity of the peer review process.

One way to effectively assess the potential for human bias is to have a sample of the same applications independently rated by multiple panels to provide a measure of inter-rater reliability.

DHS identified resource constraints as a reason for not measuring inter-rater reliability.

Appendix II: Identifying Eligible Urban Areas

As we reported in 2007, DHS first had to determine the geographic boundaries or footprint of candidate urban areas within which data were collected to estimate risk in order to determine the urban areas that were eligible to receive UASI grants,. In fiscal year 2005, the footprint was limited to city boundaries (and did not include the 10-mile buffer zone). DHS chose to further redefine the footprint for fiscal year 2006, on the basis of comments from state and local governments. DHS took several steps to identify this footprint; these included: Identifying areas with population greater than 100,000 persons and areas (cities) that had any reported threat data during that past year. For fiscal year 2006, DHS started with a total of 266 cities.

Combining cities or adjacent urban counties with shared boundaries to form single jurisdictions. For fiscal year 2006, this resulted in 172 urban areas.

Drawing a buffer zone around identified areas. A 10-mile buffer was then drawn from the border of that city/combined entity to establish candidate urban areas. This area was used to determine what information was used in the risk analysis, and represents the minimum area that had to be part of the state/urban areas defined grant application areas.

According to DHS, for fiscal year 2006, it considered other alternatives such as a radius from a city center, although such a solution created apparent inequities among urban areas. DHS incorporated buffer zones at the suggestion of stakeholders, although this action resulted in making the analysis more difficult, according to a DHS official. In addition, DHS officials told us the steps taken to determine the footprint were based on the “best fit,” as compared with other alternatives. DHS did not provide details on what criteria this comparison was based on.

A principal change between fiscal year 2007 and 2008 was the method used to identify the footprint, or boundaries, of UASI areas for the purposes of calculating relative risk. In fiscal year 2008, DHS used Metropolitan Statistical Areas (MSAs) from the Census Bureau, as required under the Implementing Recommendations of the 9/11 Commission Act of 2007.

Table 1 below provide additional information listing the urban areas by its prior geographic area captures, and the areas captured by MSAs.

Appendix III: DHS’s Model is Robust for Tier 1 UASI Areas

Population Index: Neither maximizing nor minimizing the weight of the Population Index resulted in the movement of an area into or out of Tier 1 for either FY 2007 or FY 2008.

Economic Index: In FY 2007, minimizing the weight of the Economic Index had no effect on Tier 1 placement, but increasing the weight of the Economic Index by 12.8% resulted in a new area moving into Tier 1, displacing an area that had previously been ranked in the top 7. In FY 2008, lowering the weight of the Economic Index by 15.25% resulted in a new area moving into the top 7 ranked areas, displacing an area that had been previously ranked as Tier 1, but maximizing the weight of the Economic Index had no effect on Tier 1 placement.

National Infrastructure Index: In FY 2007, maximizing the weights of the National Infrastructure Index did not result in any change in those areas designated Tier 1, but lowering the National Infrastructure Index by 5.53% resulted in a new area moving into the Tier 1 areas, displacing an area that had been previously ranked as Tier 1. In FY 2008, increasing the weight of the National Infrastructure Index by 4.68% resulted in a new area moving into the top 7 ranked areas, displacing an area that had been previously ranked as Tier 1. Similarly, lowering the National Infrastructure Index by 15% resulted in a new area moving into the Tier 1 areas.

National Security Index: In FY 2007, minimizing the weight of the National Security Index also did not result in any change in those areas designated Tier 1, but increasing the National Security Index by 7.5% resulted in a new area moving into Tier 1, displacing an area that had been previously ranked as Tier 1. In FY 2008, lowering the weight of the National Security Index by 3.73% resulted in a new area moving into the top 7 ranked areas, displacing an area that had been previously ranked as Tier 1. Increasing the National Security Index by 10% resulted in a new area moving into Tier 1, also displacing an area that had been previously ranked as Tier 1.

Urban Area Sensitivity to Changes in Consequence Index Weights is Reduced in FY 2008 for Funding Eligibility

While Tier 1 areas were similarly robust in both FY 2007 and FY 2008, the sensitivity of Tier 2 areas to changes in the weights of indices used to calculate risk scores was significant in FY 2007, but improved in FY 2008. In FY 2007, very small changes in the weights for the indices used to quantify risk for Tier 2 urban areas at the eligibility cut point resulted in changes in eligibility; however, FY 2008 results are more robust, as eligibility of urban areas is much less sensitive to changes in the index weights in the FY2008 model than it was in the FY2007 model.

Population Index: In FY 2007, decreasing the weight of the Population Index by 0.4% or increasing the weight of the Population Index by 4% resulted in one area displacing another area with regard to eligibility. However, neither maximizing nor minimizing the Population Index resulted in one area displacing another area with regard to eligibility in FY 2008.

Economic Index: In FY 2007, lowering the weight of the Economic Index by 0.24% or increasing the weight of the Economic Index by 2.4% resulted in one area displacing another area with regard to eligibility. By contrast, FY 2008 required an increase in the weight of the Economic Index by 12.33% or a decrease in the weight of the Economic Index by 10.48% resulted in one area displacing another area with regard to eligibility.

National Infrastructure Index: In FY 2007, changing the weight for the National Infrastructure Index by 1.58% (either increase or decrease) resulted in one area displacing another area with regard to eligibility, while the FY 2008 National Infrastructure Index required an increase in the weight by 5.67% or a decrease the weight by 4.54% to result in one area displacing another area with regard to eligibility.

National Security Index: In FY 2007, increasing the weight for the National Security Index by 0.08% resulted in one area displacing another area with regard to eligibility, but FY 2008 required an increase in the weight for the National Security Index by 2.34% or a decrease in the weight of the National Security Index by 1.37% to result in one area displacing another area with regard to eligibility.

Appendix IV: Contacts and Staff Acknowledgments

For further information about this statement, please contact William O. Jenkins Jr., Director, Homeland Security and Justice Issues, on (202) 512- 8777 or jenkinswo@gao.gov.

In addition to the contact named above, the following individuals also made major contributors to this report: GAO Homeland Security and Justice Issues Team—Chris Keisling, Assistant Director; John Vocino, Analyst-in-Charge; Orlando Copeland and Michael Blinde, Analysts; Linda Miller and Adam Vogt, Communications Analysts. Other major contributors to this report include: GAO Applied Methodology and Research Team—Chuck Bausell, Jr., Economist, and Virginia Chanley; and GAO Office of General Counsel—Frances Cook.