Homeland Security: DHS Risk-Based Grant Methodology Is Reasonable, But Current Version's Measure of Vulnerability is Limited
Published: 2008-06-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 2002, the Department of Homeland Security (DHS) has distributed almost $20 billion in funding to enhance the nation's capabilities to respond to acts of terrorism or other catastrophic events. In fiscal year 2007, DHS provided approximately $1.7 billion to states and urban areas through its Homeland Security Grant Program (HSGP) to prevent, protect against, respond to, and recover from acts of terrorism or other catastrophic events. As part of the Omnibus Appropriations Act of 2007, GAO was mandated to review the methodology used by DHS to allocate HSGP grants. This report addresses (1) the changes DHS has made to its risk-based methodology used to allocate grant funding from fiscal year 2007 to fiscal year 2008 and (2) whether the fiscal year 2008 methodology is reasonable. To answer these questions, GAO analyzed DHS documents related to its methodology and grant guidance, interviewed DHS officials about the grant process used in fiscal year 2007 and changes made to the process for fiscal year 2008, and used GAO's risk management framework based on best practices.
section_title: What GAO Found
paragraphs: For fiscal year 2008 HSGP grants, DHS is primarily following the same methodology it used in fiscal year 2007, but incorporated metropolitan statistical areas (MSAs) within the model used to calculate risk. The methodology consists of a three-step process--a risk analysis of urban areas and states based on measures of threat, vulnerability and consequences, an effectiveness assessment of applicants' investment justifications, and a final allocation decision. The principal change in the risk analysis model for 2008 is in the definition of the geographic boundaries of eligible urban areas. In 2007, the footprint was defined using several criteria, which included a 10-mile buffer zone around the center city. Reflecting the requirements of the Implementing Recommendations of the 9/11 Commission Act of 2007, DHS assessed risk for the Census Bureau's 100 largest MSAs by population in determining its 2008 Urban Areas Security Initiative (UASI) grant allocations. This change altered the geographic footprint of the urban areas assessed, aligning them more closely with the boundaries used by government agencies to collect some of the economic and population data used in the model. This may have resulted in DHS using data in its model that more accurately estimated the population and economy of those areas. The change to the use of MSA data in fiscal year 2008 also resulted in changes in the relative risk rankings of some urban areas. As a result, DHS officials expanded the eligible urban areas in fiscal year 2008 to a total of 60 UASI grantees, in part, to address the effects of this change to MSA data, as well as to ensure that all urban areas receiving fiscal year 2007 funding continued to receive funding in fiscal year 2008, according to DHS officials. Generally, DHS has constructed a reasonable methodology to assess risk and allocate funds within a given fiscal year. The risk analysis model DHS uses as part of its methodology includes empirical risk analysis and policy judgments to select the urban areas eligible for grants (all states are guaranteed a specified minimum percentage of grant funds available) and to allocate State Homeland Security Program (SHSP) and UASI funds. However, our review found that the vulnerability element of the risk analysis model has limitations that reduce its value. Measuring vulnerability is considered a generally-accepted practice in assessing risk; however, DHS's current risk analysis model does not measure vulnerability for each state and urban area. Rather, DHS considered all states and urban areas equally vulnerable to a successful attack and assigned every state and urban area a vulnerability score of 1.0 in the risk analysis model, which does not take into account any geographic differences. Thus, as a practical matter, the final risk scores are determined by the threat and consequences scores.

section_title: Letter
paragraphs: Since 2002, the Department of Homeland Security (DHS) has distributed  almost $20 billion in federal funding through various DHS grant programs  that provide funding to public jurisdictions and private owners/operators  for planning, equipment, and training to enhance the nation’s capabilities  to respond to terrorist attacks and, to a lesser extent, natural and  accidental disasters. In fiscal year 2007, DHS provided approximately $1.7  billion to states and urban areas through its Homeland Security Grant  Program (HSGP) to prevent, protect against, respond to, and recover from  acts of terrorism or other catastrophic events and plans to distribute  approximately $1.6 billion under this program in fiscal year 2008.
The majority of funding from the Homeland Security Grant Program is  provided through two of its five component programs: the State Homeland  Security Program (SHSP) and the Urban Areas Security Initiative (UASI).  SHSP supports building and sustaining capabilities at the state and local  levels through planning, equipment, training, and exercise activities and  helps states to implement the strategic goals and objectives included in  state homeland security strategies. SHSP provides funding to all 56 states  and territories based on a combination of assessing relative risk, and  determining the effectiveness of states’ proposed investments. UASI  addresses the unique multi-disciplinary planning, operations, equipment,  training, and exercise needs of high-threat, high-density urban areas. The  program provides funding to high-risk urban areas based on  determinations of risk and assessments of the effectiveness of the plans  for using the funds. DHS used this same risk-based methodology to  allocate $852 million in fiscal year 2008 under the Infrastructure Protection  Program, according to DHS.
The distribution of HSGP funds, including UASI funding, has raised  congressional interest about DHS’s methods in making such  determinations. For the third consecutive year, GAO has been mandated  as part of DHS’s annual appropriation to review and assess the HSGP’s  risk analysis model and risk-based allocation methodology for  determining risk and distributing funds. We responded to the mandate in  February 2008 by briefing the staffs of congressional committees on the  results of this review (see Appendix I). This report and the accompanying  appendices supplements and transmits the information provided during  those briefings.
In response to a mandate in the Consolidated Appropriations Act, 2008,  GAO reviewed the methodology used by DHS to allocate HSGP grants.  This report addresses the following questions:  1.  How has the risk-based methodology DHS uses to allocate grant  funding changed from fiscal year 2007 to fiscal year 2008?  2.  How reasonable is the fiscal year 2008 methodology?
To answer these questions, we analyzed DHS documents, including the  risk analysis models for fiscal years 2007 and 2008, grant guidance, and  presentations. To provide a basis for examining efforts at carrying out risk  management, we applied a framework for risk management that GAO  developed based on best practices and other criteria. We used our risk  management framework to examine DHS’s risk-based methodology— which includes its risk analysis model. Our analysis includes the extent to  which:   Information used in DHS’s methodology—such as specific measures  and weights—was sufficient and reliable;    Attributes of DHS’s methodology that potentially include both  government and non-government items were identified by a reasoned  process;    DHS could justify the aggregation or calculations of these attributes;    DHS documented its processes and applied written criteria when using  methods to obtain scores or weights (ie. peer review), or when ranges  or categories (ie. tiers) are used;    Relative risk rankings are sensitive to incremental changes in  assumptions or alternative perceptions related to grantee eligibility or  funding levels; and    DHS has procedures in place to update their methodology if new  information becomes available.
Finally, we interviewed DHS officials about the HSGP grant determination  process used in fiscal year 2007 and about changes made to the process  for fiscal year 2008. We performed this performance audit from September  2007 through April 2008, in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Risk management has been endorsed by Congress, the President, and the  Secretary of DHS as a way to direct finite resources to those areas that are  most at risk of terrorist attack under conditions of uncertainty. The  purpose of risk management is not to eliminate all risks, as that is an  impossible task. Rather, given limited resources, risk management is a  structured means of making informed trade-offs and choices about how to  use available resources effectively and monitoring the effect of those  choices. Thus, risk management is a continuous process that includes the  assessment of threats, vulnerabilities, and consequences to determine  what actions should be taken to reduce or eliminate one or more of these  elements of risk.
To provide a basis for examining efforts at carrying out risk management,  GAO developed a framework for risk management based on best  practices and other criteria. The framework is divided into five phases: (1)  setting strategic goals and objectives, and determining constraints; (2)  assessing the risks; (3) evaluating alternatives for addressing these risks;  (4) selecting the appropriate alternatives; and (5) implementing the  alternatives and monitoring the progress made and the results achieved  (see Fig.1).
Because we have imperfect information for assessing risks, there is a  degree of uncertainty in the information used for risk assessments (e.g.,  what the threats are and how likely they are to be realized). As a result, it  is inevitable that assumptions and policy judgments must be used in risk  analysis and management. It is important that key decision-makers  understand the basis for those assumptions and policy judgments and  their effect on the results of the risk analysis and the resource decisions  based on that analysis.
DHS has used an evolving risk-based methodology to identify the urban  areas eligible for HSGP grants and the amount of funds states and urban  areas receive (see Fig 2). For example, the risk analysis model used from  fiscal year 2001 through 2003 largely relied on measures of population to  determine the relative risk of potential grant recipients, and evolved to  measuring risk as the sum of threat, critical infrastructure and population  density calculations in fiscal years 2004 and 2005.
The fiscal year 2006 process introduced assessments of threat,  vulnerability and consequences of a terrorist attack in assessing risk. In  addition to modifications to its risk analysis model, DHS adopted an  effectiveness assessment for fiscal year 2006 to determine the anticipated  effectiveness of the various risk mitigation investments proposed by urban  areas, which affected the final amount of funds awarded to eligible areas.  For the fiscal year 2007 allocation process, DHS defined Risk as the  product of Threat times Vulnerability and Consequences, or “R= T* (V &  C)” and applied a three-step risk-based allocation methodology which  incorporates analyses of risk and effectiveness to select eligible urban  areas and allocate UASI and SHSP funds (see Fig. 3). The three steps  include:  1.  Implementation of a Risk Analysis model to calculate scores for  states and urban areas, defining relative Risk as the product of  Threat, Vulnerability and Consequences;  2.  Implementation of an Effectiveness Assessment, including a  process where state and urban area representatives acting as peer  reviewers assess and score the effectiveness of the proposed  investments submitted by the eligible applicants. This process is  also known as peer review.  3.  Calculation of a Final Allocation of funds based on states’ and  urban areas’ risk scores as adjusted by their effectiveness scores.
The Post-Katrina Emergency Management Reform Act places  responsibility for allocating and managing DHS grants with the Federal  Emergency Management Agency (FEMA).  While FEMA is responsible for  implementing the above 3-step process, FEMA relies on other DHS  components such as the National Protection and Programs Directorate  (NPPD) and the Office of Intelligence and Analysis (I&A) in the  development of the risk analysis model, which we will discuss in greater  detail below.
subsections: section_title: Risk Analysis Model
paragraphs: DHS employs a risk analysis model to assign relative risk scores to all  states and urban areas under the SHSP and UASI grant programs. These  relative risk scores are also used to differentiate which urban areas are  eligible for UASI funding. These eligible areas are divided into two tiers:  Tier 1 UASI grantees and those eligible for Tier 2. In fiscal year 2007, 45  candidates were eligible to apply for funding under the UASI program, and  eligible candidates were grouped into two tiers according to relative risk.  Tier 1 included the six highest risk areas; Tier 2 included the other 39  candidate areas. Figure 4 provides an overview of the factors that are  included in the risk analysis model for fiscal year 2007 and their relative  weights. The maximum relative risk score possible for a given area was  100. The Threat Index accounted for 20 percent of the total risk score; the  Vulnerability and Consequences Index accounted for 80 percent.
The Threat Index accounted for 20 percent of the total risk score, which  was calculated by assessing threat information for multiple years  (generally, from September 11, 2001 forward) for all candidate urban areas  and categorizing urban areas into different threat tiers. According to DHS  officials, the agency’s Office of Intelligence and Analysis (I&A) calculated  the Threat Index by (1) collecting qualitative threat information with a  nexus to international terrorism, (2) analyzing the threat information to  create threat assessments for states and urban areas, (3) empanelling  intelligence experts to review the threat assessments and reach consensus  as to the number of threat tiers, and (4) assigning threat scores. This  process, according to DHS officials, relied upon analytical judgment and  interaction with the Intelligence Community, as opposed to the use of total  counts of threats and suspicious incidents to calculate the Threat Index  for the 2006 grant cycle. The final threat assessments are approved by the  Intelligence Community—the Federal Bureau of Investigation, Central  Intelligence Agency, National Counterterrorism Center, and the Defense  Intelligence Agency—along with the DHS Under Secretary for Intelligence  and Analysis and the Secretary of DHS, according to DHS officials.
The Vulnerability and Consequences index accounts for 80 percent of the  total risk score. Because DHS considered most areas of the country  equally vulnerable to a terrorist attack given freedom of movement within  the nation, DHS assigns vulnerability a constant value of 1.0 in the formula  across all states and urban areas. Therefore, DHS’s measurement of  vulnerability and consequences is mainly a function of the seriousness of  the consequences of a successful terrorist attack, represented by four  indices: a Population Index, an Economic Index, a National Infrastructure  Index, and a National Security Index.
Population Index (40 percent). This index included nighttime  population and military dependent populations for states and urban  areas, based upon U.S. Census Bureau and Department of Defense  data. For urban areas, factors such as population density, estimated  number of daily commuters, and estimated annual visitors were also  included in this variable using data from private entities. DHS  calculated the Population Index for urban areas by identifying areas  with a population greater than 100,000 persons and cities that reported  threat data during the past year, then combined cities or adjacent  urban counties with shared boundaries to form single jurisdictions,  and drew a 10-mile buffer zone around identified areas.
Economic Index (20 percent). This index is comprised of the economic  value of the goods and services produced in either a state or an urban  area. For states, this index was calculated using U.S. Department of  Commerce data on their percentage contribution to Gross Domestic  Product. For UASI urban areas, a parallel calculation of Gross  Metropolitan Product was incorporated.
National Infrastructure Index (15 percent). This index focused on over  2,000 critical infrastructure/key resource (CIKR) assets that were  identified by DHS’s Office of Infrastructure Protection. These  particular critical infrastructure assets are divided into two rankings  that, if destroyed or disrupted, could cause significant casualties,  major economic losses, or widespread/long term disruptions to  national well-being and governance capacity. The Tier 2 CIKR assets  include the nationally-significant and high-consequence assets and  systems across 17 sectors. Tier 1 assets are a small subset of the Tier  2 list that include assets and systems certain to produce at least two of  four possible consequences if disrupted or destroyed: (1) prompt  fatalities greater that 5,000; (2) first-year economic impact of at least  $75 billion; (3) mass evacuations with prolonged (6 months or more)  absence; and (4) loss of governance or mission execution disrupting  multiple regions or critical infrastructure sectors for more than a  week, resulting in a loss of necessary services to the public.  Tier 1  assets were weighted using an average value three times as great as  Tier 2 assets.
The National Security Index (5 percent). This index considered three  key national security factors: whether military bases are present in the  state or urban area; how many critical defense industrial base facilities  are located in the state or urban area; and the total number of people  traversing international borders. Information on these inputs comes  from the Department of Defense and DHS.
section_title: Effectiveness Assessment
paragraphs: In addition to determining relative risk using the risk analysis model, DHS  added an effectiveness assessment process in fiscal year 2006 to assess  and score the effectiveness of the proposed investments submitted by  grant applicants. To assess the anticipated effectiveness of the various risk  mitigation investments that states and urban areas proposed, DHS  required states and urban areas to submit investment justifications as part  of their grant applications. The investment justifications included up to 15  “investments” or proposed solutions to address homeland security needs,  which were identified by the states and urban areas through their strategic  planning process. DHS used state and urban area representatives as peer  reviewers to assess these investment justifications. The criteria reviewers  used to score the investment justifications included the following  categories: relevance to national, state and local plans and policies such as  the National Preparedness Guidance states’ and urban areas’ homeland  security plans, anticipated impact, sustainability, regionalism, and the  applicants’ planned implementation of each proposed investment.  Reviewers on each panel assigned scores for these investment  justifications, which, according to DHS officials, were averaged to  determine a final effectiveness score for each state and urban area  applicant.
In fiscal year 2007, DHS provided states and urban areas the opportunity  to propose investment justifications that included regional collaboration  to support the achievement of outcomes that could not be accomplished if  a state or urban area tried to address them independently. States and  urban areas could choose to submit multi-state or multi-urban area  investment justifications which outlined shared investments between two  or more states or between two or more urban areas. Such investments  were eligible for up to 5 additional points on their final effectiveness score,  or up to 8 more effectiveness points for additional proposed investments,  although these additional points would not enable a state’s or urban area’s  total effectiveness score to exceed 100 points. These proposed  investments were reviewed by one of two panels established specifically  to consider multi-applicant proposals. Points were awarded based on the  degree to which multi-applicant investments showed collaboration with  partners and demonstrated value or outcomes from the joint proposal that  could not be realized by a single state or urban area.
section_title: Final Allocation Process
paragraphs: DHS allocated funds based on the risk scores of states and urban areas, as  adjusted by their effectiveness scores. DHS officials explained that while  allocations are based first upon area risk scores, the effectiveness scores  are then used to determine adjustments to states and urban areas  allocations based on an “effectiveness multiplier.” States and urban areas  with high effectiveness scores received an additional percentage of their  risk-based allocations, while states and urban areas with low effectiveness  scores had their risk-based allocations lowered by a percentage.
In addition to determining funding by risk score as adjusted by an  effectiveness multiplier, urban areas that received funds through the UASI  grant program were subject to an additional tiering process that affected  funding allocation. For example, in fiscal year 2007, the 45 eligible urban  area candidates were grouped into two tiers according to relative risk. The  Tier 2 UASI grantees included the 6 highest-risk areas; Tier 2 UASI  grantees included another 39 candidate areas ranked by risk. The 6 Tier 1  UASI grantees were allocated fifty-five percent of the available funds, or  approximately $410.8 million, while the 39 Tier 2 UASI grantees received  the remaining forty-five percent of available funds, or approximately  $336.1 million.
section_title: Shifting to Urban Area Boundaries Defined by MSA was the Primary Change to DHS’s Risk-Based Methodology in 2008
paragraphs: DHS’s risk-based methodology had few changes from fiscal year 2007 to  2008. DHS changed the definition it used to identify the UASI areas  included in the risk analysis model in 2008 from an urban area’s center city  plus a ten-mile radius to metropolitan statistical areas (MSAs) as defined  by the Census Bureau. DHS made this change in response to the 9/11Act  requirement to perform a risk assessment for the 100 largest MSAs by  population. Because the change in definition generally expanded the  geographic area of each potential UASI grant recipient, the change had an  effect on the data used to assess threat and consequences, and it may also  have resulted in the use of more accurate data in the risk analysis model.  The change to the use of MSA data in fiscal year 2008 also resulted in  changes in the relative risk rankings of some urban areas. As a result, DHS  officials expanded the eligible urban areas in fiscal year 2008 to a total of  60 UASI grantees, in part, to address the effects of this change to MSA  data, as well as to ensure that all urban areas that received fiscal year 2007  funding also received funding for fiscal year 2008, according to DHS  officials.
Changing the boundaries had an effect on the data by which risk is  calculated because the change in boundaries resulted in changes in the  population and critical assets within the new boundaries. Figure 3 below  uses the Chicago, IL urban area to illustrate this change. One benefit of the  change to MSAs was that the UASI boundaries align more closely with the  boundaries used to collect some of the economic and population data used  in the model. Consequently, the fiscal year 2008 model may have resulted  in more accurate data. Because the 2007 boundaries were based on  distance, areas inside the boundaries may have included partial census  tracts or partial counties, each of which would have required DHS to  develop rules as to how to handle the partial areas. By contrast, the MSAs  are based on counties and allow DHS to use standard census data instead  of developing an estimated population within the defined boundaries.  Additional information describing the boundaries of UASI urban areas for  fiscal year 2007 versus fiscal year 2008 is presented in Appendix II.
DHS calculated the Population Index of MSAs by: (1) using census data to  determine the population and population density of each census tract; (2)  calculating a Population Index for each individual census tract by  multiplying the census tract’s population and population density figures;  and (3) adding together the population indices of all of the census tracts  making up the MSA. DHS did not use average population density because   using an average resulted in losing information about how the population  is actually distributed among the tracts. Using averages for population  density over census tracts with dissimilar densities could have yielded  very misleading results, according to DHS officials.
The change to MSAs for fiscal year 2008 resulted in an increase of almost  162,000 square miles across the total area of urban area footprints. While 3  urban areas actually lost square mileage because of the change, the other  areas all increased their square mileage footprint by almost 2,700 square  miles on average. The increased size of urban areas’ footprints increased  the number of critical infrastructure assets that were counted within them.  We analyzed the number of Tier 1 and Tier 2 critical infrastructure assets  associated with UASI areas between fiscal year 2007 and 2008, and found a  higher number of total Tier 1 and Tier 2 critical infrastructure assets  assigned to urban areas in 2008, and–individually—almost all urban areas  increased the number of assets assigned to them.
This change to the use of MSAs also resulted in changes in urban areas  rankings, including the increase of the relative risk scores for such urban  areas as Albany, Syracuse and Rochester, NY, and Bridgeport, CT. As a  result, DHS officials expanded the eligible urban areas in fiscal year 2008  to a total of 60 with the top seven highest risk areas comprising UASI Tier  1 grantees, and the 53 other risk-ranked UASI Tier 2 grantees. As in fiscal  year 2007, the top seven UASI Tier 1 grantee areas will receive fifty-five  percent of the available funds, or approximately $429.9 million, and the  remaining 53 UASI Tier 2 grantees will receive forty-five percent of the  available funds, or approximately $351.7 million. According to DHS  officials, the decision to expand the eligible urban areas to a total of sixty  was a policy decision largely driven by two factors: the 9/11 Act  requirement that FEMA use MSAs; and the desire to continue to fund  urban areas already receiving funding.
section_title: DHS’s Risk-based Methodology is Generally Reasonable, But the Vulnerability Element of the Risk Analysis Model Has Limitations that Reduce Its Value
paragraphs: The risk-based methodology DHS uses to allocate HSGP grant dollars is  generally reasonable. It includes and considers the elements of risk  assessment—Threat, Vulnerability, and Consequences—and, as DHS’s  risk-based methodology has evolved, its results have become less sensitive  to changes in the key assumptions and weights used in the risk analysis  model. Furthermore, the indices that DHS uses to calculate the variable  constituting the greatest portion of the risk analysis model— Consequences—are reasonable. However, limitations such as the absence  of a method for measuring variations in vulnerability reduce the  vulnerability element’s value. Although DHS recognized and described the  significance of Vulnerability in its FY 2006 model, the model DHS used for  fiscal years 2007 and 2008 used a constant value of 1.0 in its formula,  rather than measuring variations in vulnerability across states and urban  areas.
subsections: section_title: DHS’s Risk Analysis Model is Reasonable Because it Contains the Key Elements of Risk Assessment, Relies on Reasonable Indices to Measure Consequences, and is Less Sensitive to Changes in Variables
paragraphs: One measure of the reasonability of DHS’s risk-based methodology is the  extent to which DHS’s risk analysis model provides a consistent method to  assess risk. Risk assessment helps decision makers identify and evaluate  potential risks facing key assets or missions so that countermeasures can  be designed and implemented to prevent or mitigate the effects of the  risks. In a risk management framework, risk assessment is a function of  Threat, Vulnerability, and Consequences, and the product of these  elements is used to develop scenarios and help inform actions that are  best suited to prevent an attack or mitigate vulnerabilities to a terrorist  attack. Threat is the probability that a specific type of attack will be  initiated against a particular target/class of targets, and analysis of threat- related data is a critical part of risk assessment. The Vulnerability of an  asset is the probability that a particular attempted attack will succeed  against a particular target or class of targets. It is usually measured against  some set of standards, such as availability/predictability, accessibility,  countermeasures in place, and target hardness (the material construction  characteristics of the asset). The Consequences of a terrorist attack  measures the adverse effects of a successful attack and may include many  forms, such as the loss of human lives, economic costs, and adverse  impact on national security. The risk analysis model used by DHS is  reasonable because it attempts to capture data on threats, vulnerabilities,  and consequences—the three types of information used in evaluating risk.
Because DHS considered most areas of the country equally vulnerable to a  terrorist attack given freedom of movement within the nation, DHS assigns  vulnerability a constant value of 1.0 in the formula across all states and  urban areas. Therefore, DHS’s measurement of vulnerability and  consequences is mainly a function of the seriousness of the consequences  of a successful terrorist attack. Because the risk analysis model is  consequences-driven, another measure of the model’s overall  reasonableness is the extent to which the indices used to calculate the  consequences component of the model are reasonable. As previously  described, the consequences component of the model is comprised of four  indices – a Population Index, an Economic Index, a National  Infrastructure Index, and a National Security Index – each assigned a  different weight. These indices are generally reasonable.
Both the population and economic indices are calculated from data  derived from reliable sources that are also publicly available, providing  additional transparency for the model. For example, according to DHS  officials, the fiscal year 2008 analysis used Gross Metropolitan Product  (GMP) estimates prepared by the consulting firm Global Insight for the  United States Conference of Mayors and the Council for the New  American City that were published in January 2007, and reported on the  GMP for 2005. In addition, the National Infrastructure Index focused on  over 2,000 Tier 1and Tier 2 critical infrastructure/key resource assets  identified by DHS’s Office of Infrastructure Protection (IP). For both fiscal  years 2007 and 2008, DHS used a collaborative, multi-step process to  create the Tier 2 CIKR list. First, IP works with sector-specific agencies to  develop criteria used to determine which assets should be included in the  asset lists. Second, these criteria are vetted with the private-sector through  sector-specific councils, who review the criteria and provide feedback to  IP. Third, IP finalizes the criteria and provides it to the sector-specific  agencies and State and Territorial Homeland Security Advisors (HSAs).  Fourth, IP asks states to nominate assets within their jurisdiction that  match the criteria. Fifth, assets nominated by states are reviewed by both  the sector-specific agencies and IP to decide which assets should comprise  the final Tier 2 list. For example, to identify the nation’s critical energy  assets, IP will work with the Department of Energy to determine which  assets and systems in the energy sector would generate the most serious  economic consequences to the Nation should they be destroyed or  disrupted. Further, in the fiscal year 2008 process, IP added a new,  additional step to allow for the resubmission of assets for reconsideration  if they are not initially selected for the Tier 2 list. In addition, the National  Security Index comprises only a small fraction of the model – 5 percent –  and has also evolved to include more precision, such as counting the  number of military personnel instead of simply the presence or absence of  military bases.  To identify the nation’s critical defense industrial bases,  the Department of Defense analyzes the impact on current warfighting  capabilities, recovery and reconstitution, threat, vulnerability, and  consequences of possible facility disruption and destruction, and other  aspects.
DHS’s approach to calculating threat, which accounts for the remaining 20  percent of the model, also represents a measure of the model’s overall  reasonableness. DHS uses analytical judgments to categorize urban areas’  threat, which ultimately determines the relative threat for each state and  urban area.  DHS has used written criteria to guide these judgments, and  DHS provided us with the criteria used in both of these years for our  review. The criteria are focused on threats from international terrorism  derived from data on credible plots, planning, and threats from  international terrorist networks, their affiliates, and those inspired by such  networks. The criteria provided guidance for categorizing areas based on  varying levels of both the credibility and the volume of threat reporting, as  well as the potential targets of threats. Results of this process are shared  with the DHS Undersecretary for Intelligence and Analysis, the FBI, and  the National Counterterrorism Center, all of whom are afforded the  opportunity to provide feedback on the placements. Additionally, DHS  develops written threat assessments that indicate whether states are  “high,” “medium,” or “low” threat states. States can provide threat  information that they have collected to DHS, but in order for that  information to affect a state’s tier placement and threat level, the  information must be relevant to international terrorism, according to DHS  officials. We reviewed several examples of these assessments from 2007,  which included key findings describing both identified and potential  threats to the state. The classified assessments addressed potential  terrorist threats to critical infrastructure in each of the 56 states and  territories.  However, DHS shared assessments only with state officials  who had appropriate security clearances.  According to DHS officials,  states without officials with sufficient clearances will receive an  unclassified version of their state’s assessment for the fiscal year 2009  grant process. DHS is also developing a process by which they can share  the threat assessments with UASI areas, including those UASI areas whose  boundaries cross state lines; however, currently the assessments are  transmitted only to the DHS state representatives and state officials, and  the states and representatives are responsible for sharing the information  with the UASI areas, according to DHS officials.
Another measure of the overall reasonableness of DHS’s risk analysis  model is the extent to which the model’s results change when the  assumptions and values built into the model, such as weights of variables,  change. A model is sensitive when a model produces materially different  results in response to small changes in its assumptions. Ideally, a model  that accurately and comprehensively assesses risk would not be sensitive,  and such a model exhibiting little sensitivity could be said to be more  robust than a model with more sensitivity to changes in assumptions  underlying the model. A robust calculation or estimation model provides  its users greater confidence in the reliability of its results.  For both fiscal  years 2007 and 2008, substantial changes had to be made to the weights of  any of the indices used in the risk model to calculate state and urban area  risk scores before there was any movement in or out of the top 7 (or Tier  1) ranked UASI areas. In other words, the model provides DHS with a level  of assurance that the highest at-risk areas have been appropriately  identified. While Tier 1 UASI areas were similarly robust in both FY 2007  and FY 2008, the sensitivity of Tier 2 UASI areas to changes in the weights  of indices used to calculate risk scores was significant in FY 2007, but  improved in FY 2008. In FY 2007, very small changes in the weights for the  indices used to quantify risk (for Tier 2 UASI areas at the eligibility cut  point) resulted in changes in eligibility; however, FY 2008 results are more  robust, as eligibility of urban areas is much less sensitive to changes in the  index weights in the FY2008 model than it was in the FY2007 model.  Appendix III provides an in-depth description of the sensitivity of the  model to specific changes in the relative weights of each index for Tier 1  and Tier 2 UASI areas.
section_title: Vulnerability Element of the Risk Analysis Model Has Limitations that Reduce Its Value
paragraphs: Although the methodology DHS uses is reasonable, the vulnerability  element of the risk analysis model—as currently calculated by DHS—has  limitations that reduce its value for providing an accurate assessment of  risk. DHS considered most areas of the country equally vulnerable to a  terrorist attack in the risk analysis model used for fiscal years 2007 and  2008 and assigned a constant value to vulnerability, which ignores  geographic differences in the social, built, and natural environments  across states and urban areas. Although DHS recognized and described the  significance of vulnerability in its FY 2006 model, the model used for fiscal  years 2007 and 2008 did not attempt to measure vulnerability. Instead,  DHS considered most areas of the country equally vulnerable to a terrorist  attack due to the freedom of individuals to move within the nation. As a  result, DHS did not measure vulnerability, but assigned it a constant value  of 1.0 across all states and urban areas.
Last year we reported that DHS measured the vulnerability of an asset type  as part of its FY2006 risk analysis. DHS used internal subject matter  experts who analyzed the general attributes of an asset type against  various terrorist attack scenarios by conducting site vulnerability analyses  on a sample of sites from the asset type in order to catalog attributes for  the generic asset. These experts evaluated vulnerability by attack scenario  and asset type pairs and assigned an ordinal value to the pair based on 10  major criteria. In describing its FY 2006 methodology, DHS acknowledged  that because all attack types are not necessarily applicable to all  infrastructures, the values for threat must be mapped against vulnerability  to represent the greatest likelihood of a successful attack. DHS also  acknowledged that vulnerability of an infrastructure asset was also a  function of many variables and recognized that it did not have sufficient  data on all infrastructures to know what specific vulnerabilities existed for  every infrastructure, what countermeasures had been deployed, and what  impact on other infrastructures each asset had. At that time, DHS noted it  would require substantial time and resource investment to fully develop  the capability to consistently assess and compare vulnerabilities across all  types of infrastructure.
Vulnerability is a crucial component of risk assessment. An asset may be  highly vulnerable to one mode of attack but have a low level of  vulnerability to another, depending on a variety of factors, such as  countermeasures already in place. According to our risk management  framework, the vulnerability of an asset is the probability that a particular  attempted attack will succeed against a particular target or class of  targets. It is usually measured against some set of standards, such as  availability/predictability, accessibility, countermeasures in place, and  target hardness (the material construction characteristics of the asset).  Each of these four elements can be evaluated based on a numerical  assignment corresponding to the conditional probability of a successful  attack. Additionally, other research has developed methods to measure  vulnerability across urban areas. For example, one study described a  quantitative methodology to characterize the vulnerability of U.S. urban  centers to terrorist attack for the potential allocation of national and  regional funding to support homeland security preparedness and response  in U.S. cities. This study found that vulnerability varied across the  country, especially in urban areas. The study noted that “place matters,”  and a one-size-fits all strategy ignores geographic differences in the social,  built, and natural environments. Furthermore, in February of 2008 the  Secretary of DHS said that “as we reduce our vulnerabilities, the  vulnerabilities change as well.” However, while earlier iterations of the  risk analysis model attempted to measure vulnerability, DHS’s risk  analysis model now considers the states and urban areas of the country  equally vulnerable to a terrorist attack and assigns a constant value to  vulnerability, which ignores geographic differences.
section_title: Conclusions
paragraphs: In fiscal year 2008, DHS will distribute approximately $1.6 billion to states  and urban areas through its Homeland Security Grant Program – a  program that has already distributed approximately $20 billion over the  past six years – to prevent, protect against, respond to, and recover from  acts of terrorism or other catastrophic events. Given that risk management  has been endorsed by the federal government as a way to direct finite  resources to those areas that are most at risk of terrorist attack under  conditions of uncertainty, it is important that DHS use a reasonable risk- based allocation methodology and risk analysis model as it allocates those  limited resources.
DHS’s risk-based allocation methodology and risk analysis model are  generally reasonable tools for measuring relative risk within a given fiscal  year, considering its use of a generally-accepted risk calculation formula;  key model results’ decreased sensitivity to incremental changes in the  assumptions related to Tier 1 UASI grantees or the eligibility for Tier 2  UASI funding, the reliability of the consequence variable component  indices, and its adoption of MSAs to calculate urban area footprints.   However, the element of vulnerability in the risk analysis model could be  improved to more accurately reflect risk.  Vulnerability is a crucial  component of risk assessment, and our work shows that DHS needs to  measure vulnerability as part of its risk analysis model to capture  variations in vulnerability across states and urban areas.
section_title: Recommendations
paragraphs: To strengthen DHS’s methodology for determining risk, we are  recommending that the Secretary of DHS take the following action:  Instruct FEMA, I&A, and NPPD - DHS components each responsible for  aspects of the risk-based methodology used to allocate funds under the  Homeland Security Grant Program - to formulate a method to measure  vulnerability in a way that captures variations across states and urban  areas, and apply this vulnerability measure in future iterations of this  risk-based grant allocation model.
section_title: Agency Comments
paragraphs: We requested comments on a draft of this report from the Secretary of  Homeland Security, FEMA, I&A, and NPPD, or their designees.  In email  comments on the draft report, FEMA and I&A concurred with our  recommendation that they formulate a method to measure vulnerability in  a way that captures variations across states and urban areas and apply this  vulnerability measure in future iterations of the risk-based grant allocation  model.  FEMA, I&A, and NPPD also provided technical comments, which  we incorporated as appropriate.
We are sending copies of this correspondence to the appropriate  congressional committees, and the Secretary of Homeland Security.
Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. For further  information about this report, please contact William Jenkins, Jr., Director,  GAO Homeland Security and Justice Issues Team, at (202)-512-8777 or at  jenkinswo@gao.gov. GAO staff members who were major contributors to  this report are listed in appendix IV.
section_title: Appendix I: Briefing for Congressional Committees, February 11-25, 2008
paragraphs: For the third consecutive year, GAO has been mandated as part of DHS’s  annual appropriation to review and assess the HSGP’s risk analysis model  and risk-based allocation methodology for determining risk and  distributing funds.  We responded to the mandate in February 2008 by  briefing the staffs of congressional committees on the results of this  review.  During the course of our engagement, we had ongoing dialog with  DHS officials regarding the extent to which written criteria were used in  the development of the Threat Index.  At that time, officials from DHS’s  Office of Intelligence and Analysis stated that the criteria were not  documented.   As a result, we noted in the accompanying presentation  slides that DHS’s approach to measuring threat did not include specific,  written criteria to use when determining the threat tiers into which states  and urban areas are placed.
As part of GAO’s agency protocols, we convened an exit conference with  DHS officials which occurred on April 14, 2008.  We provided them with a  statement of facts to reflect the information gathered during our  engagement.  At this exit conference an official from the Office of  Intelligence and Analysis said DHS had used criteria in 2007 and 2008 for  categorizing cities and states based on threat, and in further discussions  with DHS we were able to independently review these documents and  confirm that such criteria were used in the development of the Threat  Index, which is reflected in the letter above.  However, we did not modify  the accompanying presentation contained in this appendix.
subsections: section_title: Homeland Security Grant Program
section_title: Introduction
subsections: section_title: According to the Department of Homeland Security (DHS), in fiscal
paragraphs: DHS provided approximately $1.7 billion to states and urban areas  through its Homeland Security Grant Program (HSGP) to prevent,  protect against, respond to, and recover from acts of terrorism or  other catastrophic events. DHS plans to distribute about $1.6 billion  for these grants in fiscal year 2008.
The HSGP risk-based allocation process is used for the State  Homeland Security Program (SHSP) and Urban Area Security  Initiative (UASI).
In addition, DHS used this same approach to allocate $655 million in fiscal year 2007 under the Infrastructure Protection Program.
section_title: Objectives
subsections: section_title: In response to a legislative mandate and discussions with
paragraphs: relevant congressional staff, we addressed the following  questions:  1. What methodology did DHS use to allocate HSGP funds for  fiscal years 2007 and 2008, including any changes DHS  made to the eligibility and allocation processes for fiscal  year 2008 and the placement of states and urban areas  within threat tiers, and why?  2. How reasonable is DHS’s methodology?
section_title: Scope and Methodology
subsections: section_title: We analyzed DHS documents including the FY2007 and FY2008 risk analysis models,
paragraphs: grant guidance, presentations, and interviewed DHS officials about:  The HSGP grant determination process in FY07—and any changes to the   The process by which DHS’s risk analysis model is used to estimate  relative risk: Risk = Threat*(Vulnerability & Consequences);  How the effectiveness assessment process is conducted;  How final allocation decisions are made.
DHS’s methodology for ranking grantees by tiered groups and the impact of this  ranking on funding allocations.
We did our work from September 2007 and February 2008, in accordance with  generally accepted government accounting standards (GAGAS).
section_title: Background:
subsections: section_title: We’ve reviewed this program for the last 3 years. In previous
paragraphs: DHS has adopted a process of “continuous improvement” to  its methods for estimating risk and measuring applicants’  effectiveness.
Inherent uncertainty is associated with estimating risk of  terrorist attack, requiring the application of policy and analytic  judgments. The use of sensitivity analysis can help to gauge  what effects key sources of uncertainty have on outcomes.
section_title: Sensitivity of the risk analysis
subsections: section_title: In FY 2007, DHS had developed a greater understanding of the
paragraphs: sensitivity of the risk model as a result of its changes to the model.
GAO’s analysis of the FY 2007 model:It takes sizable changes to the weights of these indices used to quantify risk to change the areas that compose the Tier 1 list.
For those urban areas ranked near the bottom of Tier 2 list, very  small changes in the weights for the indices used to quantify risk  can result in changes in eligibility.
According to DHS officials, there were a number of changes in the  rankings, and these changes were driven by the required change in  FY2008 to use MSAs.
section_title: Effectiveness Assessment
paragraphs: For fiscal year 2007 DHS assessed the applications submitted by  states and eligible urban areas.
DHS used a peer-review process to assess and score the  effectiveness of proposed investments by:  Engaging the states in identifying and selecting peer   Having peer reviewers individually score investments, and  Assigning peer reviewers to panels to make final effectiveness  score determinations.
section_title: FY 2007 Effectiveness Assessment
subsections: section_title: Final Allocation Process – FY 2007 Grants Based on Both Risk and Effectiveness Scores
paragraphs: DHS allocated funds based on the risk scores of states and urban areas, as  adjusted by their effectiveness scores.
SHSP provided a minimum allocation, ensuring no state or territory’s allocation  falls below the minimum levels established by the USA PATRIOT Act.
For UASI, DHS established maximum and minimum allocation to minimize  variations in some urban areas’ final allocations between years. minimum = 0.375% of all funds appropriated for SHSP and UASI.
section_title: Final Allocation Process – Ranking UASI Grantees by Tiered groups
subsections: section_title: Fiscal year 2007, 45 eligible candidates were grouped into
paragraphs: two tiers according to relative risk.
Tiering was established from a policy judgment by DHS  leadership, according to DHS grant officials.
Tier I included the 6 highest risk areas; Tier II included the  other 39 candidate areas ranked by risk.
FY 2007 Tier I Urban Areas = 6 Urban Areas,  $410,795,000 allocated  (55 percent of available funds).
FY 2007 Tier II Urban Areas = 39 Urban Areas,  $336,105,000 allocated (45 percent of available funds).
section_title: 60 eligible UASI areas in FY 2008: Tier I = 7 highest risk areas and eligible for 55 percent of
paragraphs: available funds -- $429,896,500.
Tier II = 53 areas (14 more than FY 2007) and eligible for 45  percent of available funds -- $351,733,500.
According to DHS officials, the expansion to 60 eligible UASI  areas for FY2008 is a policy decision largely driven by two  factors:  1. The new requirement that FEMA use MSAs;  2. The desire to remain consistent with the funding.
section_title: Observations on the Reasonableness of the HSGP Grant Distribution Methodology
subsections: section_title: Observations on the Reasonableness of the HSGP Grant Distribution Methodology (continued)
paragraphs: DHS could take an additional step to evaluate the reliability and  validity of the peer review process.
One way to effectively assess the potential for human  bias is to have a sample of the same applications  independently rated by multiple panels to provide a  measure of inter-rater reliability.
DHS identified resource constraints as a reason for not  measuring inter-rater reliability.
section_title: Appendix II: Identifying Eligible Urban Areas
paragraphs: As we reported in 2007, DHS first had to determine the geographic  boundaries or footprint of candidate urban areas within which data were  collected to estimate risk in order to determine the urban areas that were  eligible to receive UASI grants,. In fiscal year 2005, the footprint was  limited to city boundaries (and did not include the 10-mile buffer zone).  DHS chose to further redefine the footprint for fiscal year 2006, on the  basis of comments from state and local governments. DHS took several  steps to identify this footprint; these included:  Identifying areas with population greater than 100,000 persons and areas  (cities) that had any reported threat data during that past year. For fiscal  year 2006, DHS started with a total of 266 cities.
Combining cities or adjacent urban counties with shared boundaries to  form single jurisdictions. For fiscal year 2006, this resulted in 172 urban  areas.
Drawing a buffer zone around identified areas. A 10-mile buffer was then  drawn from the border of that city/combined entity to establish candidate  urban areas. This area was used to determine what information was used  in the risk analysis, and represents the minimum area that had to be part  of the state/urban areas defined grant application areas.
According to DHS, for fiscal year 2006, it considered other alternatives  such as a radius from a city center, although such a solution created  apparent inequities among urban areas. DHS incorporated buffer zones at  the suggestion of stakeholders, although this action resulted in making the  analysis more difficult, according to a DHS official. In addition, DHS  officials told us the steps taken to determine the footprint were based on  the “best fit,” as compared with other alternatives. DHS did not provide  details on what criteria this comparison was based on.
A principal change between fiscal year 2007 and 2008 was the method  used to identify the footprint, or boundaries, of UASI areas for the  purposes of calculating relative risk. In fiscal year 2008, DHS used  Metropolitan Statistical Areas (MSAs) from the Census Bureau, as required  under the Implementing Recommendations of the 9/11 Commission Act of  2007.
Table 1 below provide additional information listing the urban areas by its  prior geographic area captures, and the areas captured by MSAs.
section_title: Appendix III: DHS’s Model is Robust for Tier 1 UASI Areas
paragraphs: Population Index: Neither maximizing nor minimizing the weight of the  Population Index resulted in the movement of an area into or out of Tier 1  for either FY 2007 or FY 2008.
Economic Index: In FY 2007, minimizing the weight of the Economic Index  had no effect on Tier 1 placement, but increasing the weight of the  Economic Index by 12.8% resulted in a new area moving into Tier 1,  displacing an area that had previously been ranked in the top 7. In FY  2008, lowering the weight of the Economic Index by 15.25% resulted in a  new area moving into the top 7 ranked areas, displacing an area that had  been previously ranked as Tier 1, but maximizing the weight of the  Economic Index had no effect on Tier 1 placement.
National Infrastructure Index: In FY 2007, maximizing the weights of the  National Infrastructure Index did not result in any change in those areas  designated Tier 1, but lowering the National Infrastructure Index by 5.53%  resulted in a new area moving into the Tier 1 areas, displacing an area that  had been previously ranked as Tier 1. In FY 2008, increasing the weight of  the National Infrastructure Index by 4.68% resulted in a new area moving  into the top 7 ranked areas, displacing an area that had been previously  ranked as Tier 1. Similarly, lowering the National Infrastructure Index by  15% resulted in a new area moving into the Tier 1 areas.
National Security Index: In FY 2007, minimizing the weight of the National  Security Index also did not result in any change in those areas designated  Tier 1, but increasing the National Security Index by 7.5% resulted in a new  area moving into Tier 1, displacing an area that had been previously  ranked as Tier 1. In FY 2008, lowering the weight of the National Security  Index by 3.73% resulted in a new area moving into the top 7 ranked areas,  displacing an area that had been previously ranked as Tier 1. Increasing  the National Security Index by 10% resulted in a new area moving into Tier  1, also displacing an area that had been previously ranked as Tier 1.
subsections: section_title: Urban Area Sensitivity to Changes in Consequence Index Weights is Reduced in FY 2008 for Funding Eligibility
paragraphs: While Tier 1 areas were similarly robust in both FY 2007 and FY 2008, the  sensitivity of Tier 2 areas to changes in the weights of indices used to  calculate risk scores was significant in FY 2007, but improved in FY 2008.  In FY 2007, very small changes in the weights for the indices used to  quantify risk for Tier 2 urban areas at the eligibility cut point resulted in  changes in eligibility; however, FY 2008 results are more robust, as  eligibility of urban areas is much less sensitive to changes in the index  weights in the FY2008 model than it was in the FY2007 model.
Population Index: In FY 2007, decreasing the weight of the Population  Index by 0.4% or increasing the weight of the Population Index by 4%  resulted in one area displacing another area with regard to eligibility.  However, neither maximizing nor minimizing the Population Index  resulted in one area displacing another area with regard to eligibility in FY  2008.
Economic Index: In FY 2007, lowering the weight of the Economic Index  by 0.24% or increasing the weight of the Economic Index by 2.4% resulted  in one area displacing another area with regard to eligibility. By contrast,  FY 2008 required an increase in the weight of the Economic Index by  12.33% or a decrease in the weight of the Economic Index by 10.48%  resulted in one area displacing another area with regard to eligibility.
National Infrastructure Index: In FY 2007, changing the weight for the  National Infrastructure Index by 1.58% (either increase or decrease)  resulted in one area displacing another area with regard to eligibility,  while the FY 2008 National Infrastructure Index required an increase in the  weight by 5.67% or a decrease the weight by 4.54% to result in one area  displacing another area with regard to eligibility.
National Security Index: In FY 2007, increasing the weight for the National  Security Index by 0.08% resulted in one area displacing another area with  regard to eligibility, but FY 2008 required an increase in the weight for the  National Security Index by 2.34% or a decrease in the weight of the  National Security Index by 1.37% to result in one area displacing another  area with regard to eligibility.
section_title: Appendix IV: Contacts and Staff Acknowledgments
paragraphs: For further information about this statement, please contact William O.  Jenkins Jr., Director, Homeland Security and Justice Issues, on (202) 512- 8777 or jenkinswo@gao.gov.
In addition to the contact named above, the following individuals also  made major contributors to this report: GAO Homeland Security and  Justice Issues Team—Chris Keisling, Assistant Director; John Vocino,  Analyst-in-Charge; Orlando Copeland and Michael Blinde, Analysts; Linda  Miller and Adam Vogt, Communications Analysts.  Other major  contributors to this report include: GAO Applied Methodology and  Research Team—Chuck Bausell, Jr., Economist, and Virginia Chanley; and  GAO Office of General Counsel—Frances Cook.