Quadrennial Homeland Security Review: Improved Risk Analysis and Stakeholder Consultations Could Enhance Future Reviews
Published: 2016-04-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Homeland security threats continue to evolve and include challenges ranging from terrorist attacks to natural disasters, emphasizing the need for DHS to periodically examine and strengthen the nation's homeland security strategy. Further, the Implementing Recommendations of the 9/11 Commission Act of 2007 requires DHS to conduct such a review every 4 years. GAO reported in 2011 on DHS's first QHSR and recommended DHS provide sufficient time for stakeholder consultations and examine how risk information could be used to prioritize mission efforts. DHS issued its second QHSR in 2014.
GAO was asked to assess the QHSR. This report addresses the extent to which DHS (1) examined and used risk information to inform the QHSR and its implementation, (2) aligned its budget and performance measures to QHSR mission goals, and (3) collaborated with stakeholders to develop the QHSR. GAO analyzed relevant statutes and QHSR documentation; conducted a nongeneralizable survey of DHS-identified federal and nonfederal QHSR stakeholders, receiving responses from 93 of the 182 QHSR stakeholders it contacted; and interviewed officials from DHS, federal, and nonfederal entities.
section_title: What GAO Found
paragraphs: The Department of Homeland Security (DHS) assessed risk for the second Quadrennial Homeland Security Review (QHSR) and considered threats, vulnerabilities, and consequences; however, DHS did not document how its various analyses were synthesized to generate results, thus limiting the reproducibility and defensibility of the results. Without sufficient documentation, the QHSR risk results cannot easily be validated or the assumptions tested, hindering DHS's ability to improve future assessments. In addition, the QHSR describes homeland security hazards, but does not rank those hazards or provide prioritized strategies to address them. Comparing and prioritizing risks helps identify where risk mitigation is most needed and helps justify cost-effective risk management options. Without determining prioritized risk outcomes, DHS is missing an opportunity to more efficiently mitigate risk or identify the resources required for addressing different levels and types of risks.
The President's fiscal years 2015 and 2016 budget requests for DHS were generally presented in alignment with the QHSR. However, DHS has faced challenges accounting for its spending by mission, which it is taking actions to address, such as developing a new common appropriation structure to better link the department's funding request to the execution of its missions. DHS also developed performance measures for all of the QHSR mission areas.
DHS expanded its stakeholder outreach efforts, but 43 of 61 stakeholders who provided narrative responses to one question in GAO's survey stated that collaboration with stakeholders could be improved. For example, one respondent reported that stakeholders were asked to react to information provided by DHS rather than assist in formulating the QHSR approach and execution. DHS officials reported being limited by staff, time, and other constraints, and thus directed stakeholders to provide feedback via various web-based forums. Although the online forums allowed DHS to reach 2,000 representatives during its 2014 QHSR development process, DHS's QHSR After Action Report noted that the tools were used to validate study findings instead of informing them. Without fostering interactive communication, DHS may miss opportunities to incorporate stakeholder perspectives from the entire homeland security enterprise and thereby may not have fully informed the QHSR effort.
section_title: What GAO Recommends
paragraphs: GAO is making four recommendations, including that for future QHSRs DHS improve its risk assessment documentation, prioritize risks, and ensure stakeholder meetings are interactive. DHS concurred with our recommendations.

section_title: Letter
paragraphs: Our nation faces a variety of homeland security threats—including  terrorism, cyberattacks, and natural disasters—that continue to evolve  and present an array of challenges. Given the wide range of threats  facing the nation and the multitude of governmental and non- governmental stakeholders involved in preventing and responding to  these threats, it is vital that the Department of Homeland Security (DHS)  comprehensively examine the homeland security strategy of the nation  and make changes and recommendations as necessary. The  Implementing Recommendations of the 9/11 Commission Act of 2007  (9/11 Commission Act) requires that beginning in fiscal year 2009 and  every 4 years thereafter, DHS conduct a review that provides a  comprehensive examination of the homeland security strategy of the  United States. According to the 9/11 Commission Act, the review is to  delineate and update, as appropriate, the national homeland security  strategy, outline and prioritize critical homeland security missions, and  assess the organizational alignment of DHS with the homeland security  strategy and missions. The Act further requires that DHS conduct the  quadrennial review in consultation with stakeholders, such as heads of  federal agencies; key officials of the department; state, local, and tribal  governments; private sector representatives; and academics and other  policy experts.
DHS issued its first Quadrennial Homeland Security Review (QHSR) in  February 2010, which we evaluated in two reports. We reported in  September 2011 that DHS outlined a strategic framework to guide the  homeland security efforts of the department and its partners, including  federal, state, local, and tribal government agencies; the private sector;  and non-governmental organizations. We found that time frames  provided for stakeholder consultations and outreach to nonfederal  stakeholders could be improved and that DHS did not use risk information  to inform QHSR implementation. We recommended that in future reviews,  DHS provide sufficient time for stakeholder consultations, explore options  for consulting with nonfederal stakeholders, and examine how risk  information could be considered in prioritizing QHSR implementation  mechanisms. DHS concurred with the recommendations and has taken  some actions toward addressing them, but has not fully implemented the  recommendations, as discussed later in this report. In December 2010,  we issued a report on the extent to which the QHSR addressed the 9/11  Commission Act’s required reporting elements. We reported that of the  nine QHSR reporting elements, DHS addressed three and did not fully  address six. Elements DHS addressed included a description of  homeland security threats and an explanation of underlying assumptions  for the QHSR report. Elements not fully addressed included a prioritized  list of homeland security missions and discussions of cooperation  between the federal government and state, local, and tribal governments.  We made no recommendations in the December 2010 report.
In June 2014, DHS issued its second QHSR, which described changes in  the overall security environment and refined DHS’s five homeland  security missions. In this light, you asked us to review DHS’s second  QHSR, including DHS’s process for conducting the review and for  implementing the QHSR strategy. This report addresses the following  questions:  (1)  To what extent has DHS examined and used risk information to  inform the QHSR and its implementation?  (2)  To what extent has DHS aligned its budget and performance  measures with the mission goals in the QHSR?  (3)  To what extent did DHS collaborate with stakeholders during the  2014 QHSR process?
This report also examines whether the QHSR and DHS Strategic Plan  addressed the reporting elements specified for the QHSR in the 9/11  Commission Act, which we describe in appendix I.
To determine the extent to which DHS has examined and used risk  information to inform the QHSR and its implementation, we analyzed  DHS documentation on its risk analysis process and results, such as  DHS’s Current Strategic Environment and Future Strategic Environment  reports and classified risk characterization results. We also interviewed  current and former DHS Office of Strategy, Plans, Analysis, and Risk  (SPAR) officials responsible for developing the QHSR risk analyses. We  compared DHS’s risk assessment process and documentation to the  National Infrastructure Protection Plan’s (NIPP) key characteristics of a  successful risk assessment. Specifically, we evaluated DHS’s  documentation to determine the extent to which the QHSR’s risk  assessment was complete, documented, defensible, and reproducible.  Further, we evaluated DHS’s use of risk information in the QHSR against  risk management guidance in the NIPP and DHS’s Risk Management  Fundamentals, as well as against our prior work on key characteristics for  risk assessment and management.
To determine the extent to which DHS has aligned its budget and  performance measures with the mission goals in the QHSR, we analyzed  DHS documents related to the 2014 QHSR and the Fiscal Years 2014- 2018 Strategic Plan, including: DHS Budget in Brief documents from 2015  through 2017 to determine DHS’s budget priorities since issuance of the  first QHSR; fiscal years 2012 through 2015 Future Years Homeland  Security Program (FYHSP) reports; excerpts from DHS’s fiscal years  2017-2021 and 2018-2022 resource planning guidance to determine the  extent to which budget guidance reflects the DHS missions and goals;  and DHS reports on its progress implementing a common appropriations  structure. Further, we reviewed Congressional Research Service reports  on DHS’s appropriations from fiscal year 2015 and 2016. We also  interviewed DHS officials from the Office of the Chief Financial Officer,  SPAR, and from U.S. Customs and Border Protection (CBP), U.S. Coast  Guard (USCG), Federal Emergency Management Agency (FEMA), U.S.  Immigration and Customs Enforcement (ICE), National Protection and  Programs Directorate (NPPD), and Transportation Security Administration  (TSA). We selected these components based on their share of the overall  DHS fiscal year 2015 budget, breadth of homeland security  responsibilities, and other factors. Views from these components are not  generalizable to all DHS components. We interviewed component officials  to determine the extent to which they used DHS guidance in developing  their annual budget requests and the extent to which such guidance  reflects the QHSR missions and goals. With respect to performance  measures, we reviewed DHS’s Annual Performance Plans from fiscal  years 2013-2015 and 2014-2016, reviewed our prior work on key  attributes of successful performance measures and the GPRA  Modernization Act of 2010 (GPRAMA). We also interviewed DHS officials  from SPAR and the office of Program Analysis and Evaluation (PA&E)  and from CBP, USCG, FEMA, ICE, NPPD, and TSA components in order  to determine the extent to which DHS developed performance measures  that aligned with mission goals. We compared DHS’s process for  monitoring and measuring performance to federal internal control  standards and our prior work on best practices for implementation of  strategies and initiatives.
To determine the extent to which DHS consulted with stakeholders in  developing the QHSR, we conducted a survey of 182 QHSR stakeholders  identified by DHS with a response rate of about 51 percent. The  stakeholders were identified by DHS and included representatives from  federal departments and agencies; state, local, and tribal organizations;  and DHS components, directorates and offices. DHS identified these  stakeholders based on 2014 QHSR participation. We asked open-ended  questions regarding the QHSR stakeholder consultation process,  including ways in which DHS elicited stakeholder input, how revisions to  timeframes may have impacted stakeholder responsibilities, any  challenges in collaborating with DHS and suggestions for improving future  QHSRs. Of the 93 individual stakeholders who responded to our survey,  75 respondents provided narrative comments. We analyzed these  comments to determine common benefits and challenges they identified  regarding DHS consultations during the QHSR. The comments received  from these respondents are not generalizable to the entire group of  stakeholders, but the feedback provided insights into stakeholder  perspectives on how QHSR stakeholder consultations were conducted  and how they could be improved. We compared DHS’s stakeholder  consultation efforts to project management standards.
To assess the extent to which the 2014 QHSR and Fiscal Years 2014- 2018 Strategic Plan addressed reporting elements listed in the 9/11  Commission Act, we determined the extent to which each element was  incorporated into the reports. To make this determination, three GAO  analysts independently compared the QHSR and Strategic Plan to each  of the nine reporting elements to determine whether each element was  addressed, not fully addressed, or not addressed. In cases when the  analysts disagreed, they reviewed and discussed their independent  assessments to reach concurrence. In addition, we interviewed DHS  officials involved in the quadrennial review to determine DHS’s position  on how DHS addressed the 9/11 Commission Act reporting requirements.
We conducted this performance audit from January 2015 to April 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. See appendix II for a more  detailed discussion of our objectives, scope, and methodology.
subsections: section_title: Background
subsections: section_title: The 9/11 Commission Act Establishes QHSR Reporting Elements
paragraphs: Section 707 of the Homeland Security Act of 2002, as amended by the  9/11 Commission Act, requires a review of the nation’s homeland security  every four years. Under the 9/11 Commission Act, DHS is responsible  for developing the QHSR and for submitting a report that is to include  multiple elements, such as: a description of the threats to the assumed or  defined national homeland security interests of the United States; the  national homeland security strategy, including a prioritized list of the  critical homeland security missions of the United States; an assessment  of the organizational alignment of DHS with the applicable national  homeland security strategy and the homeland security mission areas  outlined; and a discussion of the status of cooperation among federal  agencies in the effort to promote national homeland security, among other  elements.
section_title: DHS’s 2014 QHSR Process
paragraphs: The Department initiated the second QHSR in summer 2012. Led by  SPAR within the Office of Policy, the department reviewed the homeland  security strategic environment as well as the DHS missions and goals. In  2013, DHS issued its terms of reference outlining the framework for  conducting the second quadrennial review and identified threats and  assumptions to be used in conducting the review. The QHSR was  conducted in four phases, as shown in figure 1, and released in June of  2014.
Based on the assessments conducted in the preparation phase, DHS  leadership directed five QHSR studies, as shown above. Each study  group was composed of officials from across DHS offices and  components, led by a DHS SPAR official, and facilitated by an  independent subject matter expert from the Homeland Security Studies  and Analysis Institute. These study groups conducted their analysis  over a 6 month period and shared their work products, such as outline  missions and assumptions, with other stakeholder groups in order to  develop goals and objectives for each homeland security mission. At the  end of the study group period, DHS senior leadership, including the  Deputy Secretary of Homeland Security, the Office of General Counsel,  and office and component heads, met multiple times to review and  discuss the study group recommendations. DHS SPAR consolidated the  study groups’ recommendations into a draft QHSR report and obtained  and incorporated feedback on the draft report from other federal agencies  and stakeholder groups, including stakeholders listed in the 9/11  Commission Act with which DHS was to consult in conducting the 2014  QHSR review.
DHS solicited input from various stakeholder groups in conducting the  second QHSR. The 9/11 Commission Act specifically required DHS to  consult with the heads of seven federal agencies. DHS consulted with  these agencies and also sought input from a range of other stakeholders,  including its directorates, offices, and components; other federal  departments and agencies; and nonfederal governmental and  nongovernmental entities and representatives, such as state, local and  tribal governmental associations, members of Congress, private sector  representatives, academics, and other policy experts. DHS SPAR  consulted with the Office of Intergovernmental Affairs and the Office of  State and Local Law Enforcement to identify and contact relevant  stakeholder organizations.
section_title: DHS Risk Management Framework
paragraphs: We, Congress, the 9/11 Commission, and others have recommended that  federal agencies with homeland security responsibilities use a risk  management approach to help ensure that finite resources are dedicated  to assets or activities considered to have the highest security priority. In  its Risk Management Fundamentals and other guidance, DHS articulates  a risk management framework to guide its homeland security planning,  preparation, and execution of its missions. DHS states that risk  management is the process for identifying, analyzing, and communicating  risk and accepting, avoiding, transferring, or controlling it to an acceptable  level. The National Infrastructure Protection Plan (NIPP) also defines  DHS’s risk management framework and includes key principles for an  effective risk assessment. Specifically, a risk assessment should be  complete, documented, reproducible, and defensible, as defined in   table 1.
section_title: DHS Budget Formulation Process
paragraphs: DHS uses a planning, programming, budgeting, and execution (PPBE)  process to allocate resources. DHS’s PPBE process produces the multi- year funding plans presented in the Future Years Homeland Security  Program (FYHSP). According to DHS guidance, at the outset of the  annual PPBE process, the department’s Office of Policy and the Chief  Financial Officer (OCFO) should provide planning and fiscal guidance,  respectively, to the department’s 16 component agencies. In accordance  with this planning and fiscal guidance, the components should produce 5- year funding plans—called Resource Allocation Plans—that are  submitted to the CFO and reviewed by DHS’s senior leaders. DHS senior  leadership is expected to modify the plans in accordance with their  priorities and assessments and submit them to the Office of Management  and Budget (OMB), which uses the plans to inform the President’s annual  budget request. DHS guidance establishes approximate timelines for  when guidance is provided to components and when budget plans are  due, as shown below in figure 3.
section_title: DHS Mission Goals and Performance Measures
paragraphs: The 2014 QHSR outlines the same five missions set forth in the first  QHSR in 2010, but the 2014 review refined the missions to reflect  evolving threats. The five missions are to (1) Prevent Terrorism and  Enhance Security, (2) Secure and Manage Our Borders, (3) Enforce and  Administer Our Immigration Laws, (4) Safeguard and Secure  Cyberspace, and (5) Strengthen National Preparedness and Resilience.  Changes to the DHS missions were primarily in Mission 4 to Safeguard  and Secure Cyberspace. Specifically, the 2014 QHSR provides enhanced  goals for cybersecurity protection that include leveraging technology and  enhancing investigative capabilities, as shown in figure 4.
DHS assesses and reports on progress for the 2014 QHSR mission goals  in its Annual Performance Report, which provides suites of performance  measures that assess progress for each QHSR mission goal area. The  Annual Performance Report includes performance targets as well as  information on agency priority goals, initiatives, and challenges. The  performance results identified in the report are organized around the  missions and goals outlined in the DHS Strategic Plan.
The DHS Fiscal Year 2014-2018 Strategic Plan focuses on how DHS  plans to implement the goals laid out in the 2014 QHSR. Specifically, the  Strategic Plan describes the QHSR homeland security missions and  goals and lists strategies to achieve the goals. For example, for Mission 1  (Prevent Terrorism and Enhance Security), to achieve Goal 1.1 (prevent  terrorist attacks), one strategy listed in the Strategic Plan is to analyze,  fuse, and disseminate terrorism information by sharing it with  stakeholders across the homeland security enterprise. In addition, for  Mission 4 (Safeguard and Secure Cyberspace), to achieve Goal 4.2  (secure the federal civilian government information technology  enterprise), one strategy listed in the Strategic Plan is to ensure  government-wide policies and standards are consistently and effectively  implemented. See figure 5 for a list of additional strategies linked to  homeland security missions and goals.
Move mouse over goal to see the objectives. For a noninteractive version, see appendix IV.
section_title: Better Documentation of the QHSR Risk Assessment Process and Prioritization of Results Could Strengthen DHS Risk Management Efforts
subsections: section_title: DHS Sought to Identify Strategically Significant Risks for the 2014 QHSR
paragraphs: DHS conducted a risk assessment for the second QHSR, which it  referred to as the Homeland Security Strategic Environment Assessment  (HSSEA). According to DHS, the purpose of the HSSEA was to  characterize those risks, threats, current and future trends, and critical  uncertainties that will most affect homeland security in the 2015 to 2019  timeframe. The HSSEA is the product of several different assessments  that DHS developed for this effort, including the Current Strategic  Environment, Future Strategic Environment, Homeland Security Planning  Threat Assessment, Homeland Security National Risk Characterization  (HSNRC), and the System Mapping Initiative. All combined, DHS  intended these documents to inform the QHSR’s narrative describing the  homeland security strategic environment as well as informing the  selection of priority strategic issues to study during the 2014 QHSR,  among other goals as shown in figure 6.
section_title: DHS Did Not Incorporate All Elements of a Successful Risk Assessment in the 2014 QHSR
paragraphs: For the HSSEA, DHS evaluated threat information and assessed relative  likelihood and consequences across various hazards that DHS  determined to be significant, including intentional acts of terrorism, natural  hazards, and technological accidents and infrastructure failures; however,  DHS did not incorporate other key elements of a successful risk  assessment into its risk assessment methodology. Specifically, DHS did  not clearly document how its various analyses were synthesized to  generate risk results; did not design an assessment that could produce  comparable, repeatable results; and did not communicate the implications  of any uncertainty in the results. DHS’s performance in each NIPP  element—complete, documented, reproducible, defensible—is discussed  in further detail below.
subsections: section_title: Complete
paragraphs: The HSSEA generally follows the NIPP standard for a complete risk  assessment in that it assesses consequence, vulnerability, and threat.  Regarding threat information, DHS officials said that they relied on two  primary sources—the classified Homeland Security Planning Threat  Assessment, developed by DHS Intelligence and Analysis (I&A), and the  publically available 2012 Worldwide Threat Assessment of the US  Intelligence Community, which is prepared by the Office of the Director of  National Intelligence (ODNI). The ODNI assessment discusses the rapid  changes in the threat environment which ODNI states is now more  diverse and interconnected than any other time in history. DHS officials  said that they used the unclassified assessment as well as select  assessments from the I&A Homeland Security Planning Threat  Assessment so that they did not face classification challenges when  sharing information with partners. DHS I&A reported using I&A and  intelligence community data, as well as data from component-level  intelligence units. Further, the ODNI-developed assessment had been  vetted through the intelligence community, which DHS officials said  added to its legitimacy.
Regarding consequence and vulnerability, DHS primarily assessed these  two factors in the Homeland Security National Risk Characterization  (HSNRC), where it evaluated the likelihood and potential consequences  of 40 hazards ranging from acts of terrorism to hurricanes. DHS  compared the impacts of these hazards according to their direct  economic impact, loss of life, and potential for injuries and illnesses in  order to identify what the department referred to as standout risks—risks  that DHS believed would result in substantial negative consequences— and to allow decision makers to evaluate risk management alternatives in  terms of their effect on increasing or decreasing risk. Additionally, DHS  sought to increase the usefulness of the QHSR risk assessment by  including an analysis of current and future strategic trends into its broader  analysis. Specifically, DHS officials said that the System Mapping  Initiative, 2012 Current Strategic Environment, and 2013 Future Strategic  Environment reports provided important insights on trends and other  factors for the QHSR and resulted in what DHS referred to as a dynamic  risk assessment.
section_title: Documented
paragraphs: Documented  The methodology and the assessment must  clearly document what information is used  and how it is synthesized to generate a risk  estimate. Any assumptions, weighting factors,  and subjective judgments need to be  transparent to the user of the methodology, its  audience, and others who are expected to  use the results. The types of decisions that  the risk assessment is designed to support  and the timeframe of the assessment (e.g.,  current conditions versus future operations)  should be given.
As mentioned previously, the HSSEA is the product of several different  assessments and was intended to be dynamic—in that, according to  DHS, it captures possible future scenarios and can help decision-makers  be more strategic—but DHS did not fully document how the findings and  results of the various assessments were synthesized to generate the  QHSR risk results, among other issues.
For the HSNRC—where DHS ranked various hazards according to their  likelihood and potential consequences—DHS did not fully document how  its results (from both its quantitative and qualitative analyses) were  synthesized to generate its list of standout hazards—a key element of the  NIPP standards for documentation. Specifically, in the documentation  provided by DHS, we identified DHS’s steps for conducting the HSNRC,  the hazard scope, consequence categories, and methodological caveats.  In separate classified documentation, we reviewed a rank-ordering of  those hazards that DHS SPAR officials said resulted from its HSNRC  analysis. However, DHS’s documentation included only a limited  description of how and what data DHS relied on to generate its standout  hazards, and no description of how any data or methodological limitations  should be interpreted by users of the results.
Additionally, DHS’s documentation included only limited information on  data sources and the specific roles of subject matter experts. For  example, regarding data sources, DHS cited “historical record” and  “subject matter expert judgment,” without providing further detail on what  record or judgements were relied upon. In classified Risk Summary  Sheets, DHS officials reported documenting additional detail about each  hazard, including the data sources and likelihood and consequence  estimates; however, this documentation does not clarify how the  information was synthesized to generate the final risk analysis results. In  addition, DHS reported that in light of limited available data, it relied on  the expertise of subject matter experts; however, DHS did not document  how these subjective judgements were weighed against other available  data to generate results, and it is unclear how much or how little the  subject matter experts’ opinions were the primary basis for DHS’s  findings. In a 2010 review of DHS’s approach to risk assessment, the  National Academies also found that DHS had not documented how  subject matter expert assumptions were made, which limits the ability to  test the validity of those assumptions, bring to light dissenting views of  experts, or inform future updates and improvements to the risk models.
Last, DHS did not document or provide an explanation of how all of the  various HSSEA elements—the system maps, the risk ranking from the  HSNRC, the trends analysis, or the threat report—were combined to  generate the QHSR risk narrative and other risk findings. According to the  NIPP, sufficient documentation can help improve or modify a risk  methodology so that the investment and expertise they represent can be  used to support risk management activities. DHS provided an explanation  that its various analyses were combined in order to generate insights, but  no other description of the weighting factors, assumptions, and subjective  judgments that went into synthesizing static and dynamic risk in the  HSSEA. DHS did, however, describe the types of decisions the HSSEA is  designed to support, as called for in the NIPP. Specifically, the HSSEA  should, according to DHS, reveal blind spots, place homeland security  events into a broader context, and identify key challenges and  opportunities, among other goals. However, beyond informing a strategic  context, it is unclear how decision-makers should use and interpret the  HSSEA results.
section_title: Reproducible and Defensible
paragraphs: Reproducible  The methodology must produce comparable,  repeatable results, even though assessments  of different critical infrastructure and key  resources may be performed by different  analysts or teams of analysts. It must  minimize the number and impact of subjective  judgments, leaving policy and value  judgments to be applied by decision makers.  Defensible  The risk methodology must logically integrate  its components, making appropriate use of the  professional disciplines relevant to the  analysis, as well as be free from significant  errors or omissions. Uncertainty associated  with consequence estimates and confidence  in the vulnerability and threat estimates  should be communicated.
DHS’s lack of documentation also limits the reproducibility and  defensibility of the results, since the assessment cannot easily be  validated or the assumptions tested, further hindering DHS’s ability to  improve future iterations of the assessment. Specifically, in the HSNRC— the element of the HSSEA that seeks to generate risk rankings—DHS  evaluates the likelihood and consequences of 40 hazards that cover  terrorism, natural hazards, and technological accidents or infrastructure  failures. However, given the level of uncertainty cited regarding the  analysis of these hazards, it is unclear how DHS determined their risk- based rank-order, as well as what findings could be drawn from the  results. For example, with available data, DHS found that it could only  compare consequences across three consequence categories—fatalities,  injuries and illnesses, and direct economic loss. The specific definitions  for injuries and illness as well as direct economic loss vary across the  threats and hazards, and DHS caveated its findings by noting that some  hazards used different definitions of economic consequences, which  resulted in substantially different levels of consequences, thus limiting the  hazards’ comparability and making it difficult to determine their relative  severity. Last, DHS caveated its findings by acknowledging that there are  large bands of uncertainty, especially for those estimates that rely on  subject matter expert judgement. Although DHS stated these and other  limitations in its supporting documentation and on the presentation of its  results, DHS did not communicate any implications this uncertainty would  have on use and interpretation of the results except to say that  simultaneously considering multiple consequence categories makes  ascertaining the top risks very difficult.
Despite these limitations, DHS did provide a clear explanation of its  guiding principles for selecting the HSNRC hazard scope, which included  national-level significance, relevance to DHS, relevance to the 2014  QHSR, and availability of data. Further, DHS provided definitions of all  the hazards for which it evaluated frequencies and consequences, as well  as which consequences it considered. DHS also explained that it  evaluated hazard frequencies using the USCG’s National Maritime  Security Risk Assessment frequency categories (which looks at  frequencies ranging from over 550 events per year to 1 event every 180  or more years). Last, the technical working group that developed the  HSNRC provided several recommendations for future HSNRC work and a  list of next steps, which included developing a departmental risk modeling  and analytical resource toolbox.
DHS officials said that conducting and documenting a risk assessment of  this scale was a complex undertaking which was limited by both time and  available personnel and that finding comparable or reliable data was  challenging. Officials said that the QHSR risk assessment process was  continually evolving and the assessment team was learning and adapting  as they conducted the analysis. Nonetheless, in its Risk Management  Fundamentals, DHS states that those affected by a risk management  approach should be able to understand how results were derived and the  process for doing so. Further, the NIPP provides guidance on how to  document a risk assessment methodology and key information—such as  the implications of uncertainty in the estimates, such as uncertainty in the  number of fatalities or dollars of economic loss—that should be  communicated to users of the risk results. Moreover, without sufficient  documentation of its HSSEA risk methodology, DHS is hindered in its  ability to reproduce the assessment for future QHSRs and limits the  defensibility of the results. Incorporating the key principles of a successful  risk assessment methodology can help DHS demonstrate the basis of its  risk management strategies and help ensure their validity for guiding  QHSR missions and goals.
section_title: QHSR Describes Various Risks Facing the Nation, but Prioritization of Risks Could Enhance DHS Risk Management Efforts
paragraphs: In the QHSR, DHS described a range of strategic risks—from the  evolving terrorist threat to the impacts of aging critical infrastructure  systems—but did not compare or prioritize risks to identify where  mitigation efforts are most needed, as called for in the NIPP. Specifically,  DHS describes the homeland security hazards it assessed as (1)  strategic priorities, (2) the key areas of change (which DHS refers to as  the threats and hazards to the nation), (3) the prevailing challenges that  pose the most risk, and (4) areas of ongoing priority and emphasis, as  shown in figure 7.
For the QHSR risk analysis, DHS officials said they not only wanted to  develop a homeland security risk characterization, but to augment this  information to provide more useful tools for DHS-wide and component  decision making. However, despite DHS’s goals for the QHSR  assessment, officials from the six components we met with (CBP, USCG,  FEMA, ICE, NPPD, and TSA) stated that the risk results reported in the  QHSR did not affect their component’s resource allocation, operational  planning, or strategic planning decisions. In addition, officials from 5 of  the 6 components reported not using any of the stand-alone QHSR risk  products in their own risk assessment or strategic planning efforts.  Component officials reported that the QHSR risk results were highly  generalized and encompassed essentially all of DHS’s missions—in other  words, they did not offer priorities or a sense of relative risks. Component  officials added that since the QHSR results were not specific or granular,  they generally relied on their own analyses when making strategic,  operational, or resource planning decisions. For example, FEMA officials  agreed with the QHSR insights that climate change and natural disasters  were a serious threat to homeland security; however, officials said they  have been monitoring these issues through FEMA’s multi-year Strategic  Foresight Initiative. Further, officials provided several reasons why they  do not use or are unable to use the stand-alone risk products. For  example, officials from one component questioned the quality of the  threat information and said they would like to see a more specific ranking  of threats, while officials from another component said the results were  too broad. Finally, officials from 3 of the 6 components were unaware of  the stand-alone risk documents.
DHS officials stated that the QHSR presented a list of key homeland  security areas and explained that they chose not to rank or prioritize  within those areas because of concerns on the part of senior leadership,  among other issues. For instance, former DHS officials who led the  development of the HSSEA said that there were some hazards that stood  out as high risk across multiple consequence categories, but reported  DHS did not rank the risks because they were unable to develop criteria  for risk tiers (such as what would be high risk, medium risk, or low risk, for  example). Further, officials were concerned that risk-ranked priorities  might imply the need to make resource changes to particular areas, when  risk is just one of many inputs for decision-making.
Former DHS officials who managed the QHSR risk assessment also said  that conducting a DHS-wide risk characterization was a challenge on  many levels. It required substantial dedicated resources and consensus  on what the analysis would describe (e.g., which scenarios and models  should be used). The officials added that there are not enough resources  at DHS to put sufficient effort into an enterprise-wide risk assessment that  includes prioritization of risk outcomes. Further, the officials  acknowledged that risk communication—including communicating the  process for and implications of prioritized risk-rankings—is a challenge.  Officials said that integrating risk-informed decision-making throughout  the department will require continued growth of DHS’s risk analysis  community. The National Academies 2010 review of DHS’s approach to  risk management echoed these statements in their recommendations to  DHS, which included building a strong risk culture at DHS and  incorporating robust social science capabilities into risk analyses and risk  management practices.
DHS officials said that they used the HSSEA results and QHSR to  broadly inform department-level strategic planning. For example, officials  said the System Mapping Initiative helped inform the development of the  Southern Border and Approaches Campaign Plan and Strategy.  Officials added that the risk results helped inform the selection of strategic  priorities in the DHS Strategic Priorities Framework and were  incorporated into the annual budget guidance DHS provides to  components. Because the Strategic Priorities Framework and annual  budget guidance are considered internal deliberative work products, we  were unable to review the complete documents in order to determine the  extent to which the HSSEA and QHSR risk results were included.  However, officials from ICE reported that the Strategic Priorities  Framework was useful for understanding DHS’s position on where to  accept risk.
The 2013 risk management supplement to the NIPP states that  comparing and prioritizing the risks faced by different entities helps  identify where risk mitigation is most needed and determines and helps  justify the selection of the most cost-effective risk management options.  This risk prioritization supports resource allocation decisions, guides  investments in these programs, and highlights the measures that offer the  greatest return on investment. According to the NIPP, the risk  prioritization process involves aggregating, combining, and analyzing risk  assessment results to determine which assets, systems, networks,  sectors, or combinations of these face the highest risk so that risk  management priorities can be established. It also provides the basis for  understanding potential risk-mitigation benefits that are used to inform  planning and resource decisions. The QHSR risk assessment describes a  wide range of homeland security challenges and is a valuable step toward  using risk information to prioritize and select risk management activities.  While the QHSR makes use of risk information, it provides limited  information to support resource allocation or risk-based strategic  planning, such as a relative risk ranking of the department’s strategic  priorities. Without determining and communicating prioritized risk analysis  outcomes, DHS is missing an opportunity to more efficiently implement  programs, strategies, and policies to mitigate risk and is limited in its  ability to identify the resources required for addressing different levels and  types of risks.
section_title: DHS Budget Request and Performance Measures Align with QHSR Mission Goals
subsections: section_title: DHS’s Budget Request Aligns with the QHSR, and DHS is Taking Actions to Account for its Spending by Mission Area
paragraphs: The Department’s annual budget request was generally presented in  alignment with the QHSR mission areas, but DHS has faced challenges  accounting for its spending by mission, which it is taking actions to  address. Specifically, the manner in which DHS’s annual appropriation is  structured does not directly track to the DHS missions, and DHS officials  reported that significant budget disparities and inconsistencies between  its components and their appropriations and programs have contributed  to a lack of transparency, inhibited comparisons between programs, and  complicated managerial decision making.
The President’s annual budget request for the department, as reflected in  the DHS’s Budget in Brief for fiscal years 2015 and 2016, generally  described funding priorities and requests at an organizational level (such  as by component, office, and directorate), not by QHSR mission area.  The Budget in Brief does, however, highlight select programs and  activities that fall within each of the department’s QHSR mission areas.  For example, the fiscal year 2016 Budget in Brief provided that within  QHSR mission area 1—Preventing Terrorism and Enhancing Security— the department requested $3.7 billion for TSA screening operations, $101  million for radiological and nuclear detection equipment acquisition, and  $94.5 million for Infrastructure Security Compliance funding to secure  high risk chemical facilities, among other requests. The fiscal year 2016  Budget in Brief provided similar examples within each DHS mission area,  as well as examples of requests for other departmental initiatives.
In general, DHS receives its funding through the enactment of an annual  appropriations act. Historically, DHS’s annual appropriations have  generally been organized into five titles, within which over 70 individual  appropriation accounts and over 100 programs, projects, and activities  (PPAs) are funded. The PPAs within DHS’s appropriation are generally  not organized by mission-based programs, and according to DHS officials  do not afford transparency or an easy understanding of how federal funds  are being used. To provide increased visibility, comparability, and  information on which to base resource allocation decisions, the report of  the Committee on Appropriations of the House of Representatives  accompanying the Department’s fiscal year 2015 appropriations bill  directed DHS to develop a common appropriation structure (CAS) for  implementation in the fiscal year 2017 budget cycle. According DHS’s  fiscal year 2017 Budget in Brief, the President’s budget for the  Department was submitted under a CAS that will improve DHS’s  planning, programming, budgeting, and execution processes through a  common framework consolidating its over 70 appropriations accounts into  four standard appropriations and will better align programs to the DHS  strategic missions and goals.
Officials from the Office of the Chief Financial Officer (OCFO) explained  some of the potential benefits they hope to achieve through the CAS. For  example, OCFO officials explained that while under the current system  DHS is able to report broadly—such as on the amount spent on border  security—it cannot report on specific obligations and expenditures within  that category because the funding spans multiple agencies with different  accounting systems. However, the CAS (in conjunction with another  ongoing project called the Accounting Classification Structure), is  intended in part to help DHS better bridge its budget requests with  execution of the QHSR strategy. Specifically, officials said their top level  of accounting will reflect the life cycle of funds, while the second level will  be standardized to newly defined mission programs. Officials said this  change will make it easier to understand and compare what is currently  captured by 100-plus PPAs that do not have a direct link to the QHSR  missions and goals. DHS currently tracks this information to some extent,  but in what they referred to as a shadow system that is not sustainable  over the long term.
Although DHS expects the CAS to improve accountability of funds and  better align its PPA structure to focus on mission programs, DHS has  reported that migrating over 70 appropriations accounts to four basic  appropriation fund types will entail changes to the department’s PPA  structure and that DHS has already faced challenges upgrading and  integrating the many varied DHS financial systems, as well as maintaining  oversight over the funding. DHS budget officials stated that DHS is  working to manage these and other challenges through regular  communication with component-level leadership and by briefing Congress  regarding the budget and staffing levels needed to successfully complete  the project.
section_title: DHS Budget Guidance Reflects QHSR Priorities; Providing Guidance to Components Sooner is an Ongoing Priority
paragraphs: DHS has taken action to improve its guidance to components for their  annual budgeting processes—one of many actions underway as part of  the department’s Unity of Effort initiative. Officials from each component  we interviewed (CBP, USCG, FEMA, ICE, NPPD, TSA) said the  department’s annual Resource Planning Guidance (RPG) mirrored the  priorities listed in the QHSR and fiscal year 2014-2018 Strategic Plan.  Further, officials from 5 of 6 components said that the RPG was an  improvement over the previous budget guidance, known as the Integrated  Planning Guidance, since the RPG’s development was more transparent  and the priorities were clearer. However, officials from 5 of the 6  components said the most recent guidance—for the fiscal years 2017- 2021 budget cycle—was not issued with sufficient time to be incorporated  into the budgeting process, which requires components to submit their  budgets to DHS in the spring of each year. Specifically, officials said they  did not receive the guidance until February 2015 for an April 2015  deadline. Officials said that this potentially limited their ability to staff issue  teams to explore budget issues and to engage in more detailed planning  and analysis. Component officials reported that receiving the guidance in  the fall would be preferable. DHS policy officials are aware of this concern  and issued its fiscal years 2018 through 2022 guidance in December  2015 and stated that the department plans to issue future guidance to  components before the end of each calendar year.
section_title: DHS Implemented Performance Measures for Each QHSR Mission and Goal
paragraphs: DHS developed performance measures that link with each of the sixteen  2014 QHSR mission goals, as shown in table 2, thus meeting one of the  attributes of successful performance measures—that they should link, or  align, with an agency’s goals and missions.
We reported in 2011 that DHS had not developed performance measures  for all of the 2010 QHSR missions, goals, and objectives. Specifically,  DHS had developed performance measures for 13 of the 14 mission  goals, having not yet developed performance measures for Goal 2.3,  Disrupt and Dismantle Transnational Criminal Organizations. For the  2014 QHSR, as mentioned above, the number of mission goals increased  to 16 (instead of 14), and DHS had established performance measures  for all 16 mission goals—including Goal 2.3—as well as new measures  related to new Mission 4 goals for cybersecurity. DHS’s Annual  Performance Report, which presents DHS’s performance measures and  applicable results aligned to missions, and provides the planned  performance targets and priority goals, indicated that at the end of fiscal  year 2014, 63 percent of all of DHS’s measures met their targets, and a  review of trends in the results showed that 71 percent of the measures  sustained or improved performance from fiscal year 2013. However,  according to DHS, not all of the mission goals have credible outcome  measures. We did not assess the quality or usefulness of the  performance measures for the 2014 QHSR mission goals, as this was  outside the scope of our review.
According to DHS SPAR and PA&E officials, both the 2010 and 2014  QHSR reports informally relied on feedback from DHS performance  measures to inform changes to the QHSR. In addition, DHS has  developed an approach relying on an existing process—referred to as the  Strategic Review Process—that assesses whether changes are needed  to broad DHS planning documents, such as the QHSR, based on  performance measure results. In 2014, DHS conducted its first Strategic  Review Process while the second QHSR was in its final clearance and  publication phase. According to DHS officials, findings from the Strategic  Review Process may help provide feedback for areas of focus and new  measure development. Further, according to officials, the use of the DHS  Annual Performance Report is just one of multiple resources DHS SPAR  will use to inform development of the next QHSR.
section_title: DHS Expanded Outreach to Stakeholders but Could Improve Collaboration
subsections: section_title: DHS Sought QHSR Input from a Broader Range of Stakeholders
paragraphs: DHS took steps to expand its outreach in 2014 QHSR process to elicit  input from a broader set of stakeholders, including additional nonfederal  entities such as state, local, and tribal governments; private organizations  and foreign governments. In addition to the seven federal agencies  specified by the 9/11 Commission Act, DHS consulted directly with two  additional federal agencies and doubled the number of stakeholder  organizations that it contacted during the 2014 QHSR process. DHS  also provided briefings to five foreign governments—Australia, Canada,  Mexico, New Zealand, and the United Kingdom—and provided regular  briefings to members of Congress, the Executive Office of the President,  and other federal partners. In total, DHS estimates that it contacted about  40,000 homeland security stakeholders via email requesting their input  and expertise in the 2014 QHSR process. Although DHS SPAR officials  used multiple mechanisms to reach stakeholders during the 2014 QHSR  process, including briefings, meetings and existing committee structures,  it relied primarily on its electronic tools to elicit input from nonfederal  stakeholders. Figure 8 provides an overview of the collaboration  mechanisms used to engage stakeholders.
According to DHS officials, the electronic tools elicited the views of more  than 2,000 registered participants and informed the analytical studies that  ultimately shaped the final QHSR report. Given the volume and diversity  of views among the stakeholders, DHS determined that using the  electronic tool mechanism was the most efficient way to capture the  majority of the input from the nonfederal entities for the 2014 QHSR  review.
section_title: Time Constraints Present in First QHSR Persisted in Second QHSR
paragraphs: In 2011, we recommended that DHS provide more time for consulting  with stakeholders during the QHSR process to help ensure that  stakeholders have the time needed to review QHSR documents and  provide input into the review. We recommended that DHS build this time  into the department’s project planning for the 2014 QHSR. Yet, this  recommendation was not implemented and the timeliness issue persisted  in the 2014 QHSR, based on what QHSR stakeholders told us. In a  survey we conducted of QHSR stakeholders, we asked whether DHS  provided a clear overview of the timeframes and stakeholder  responsibilities during each of the four phases of development.  Stakeholders varied in their responses from about fourteen percent  reporting that they did not receive a clear overview of the timeframes and  responsibilities in phase 1 to 34 percent in phase 4 of the QHSR process.  Stakeholder narrative responses also indicate that these timeframes  shifted and that overall time constraints persisted throughout the QHSR  process. For example, in the narrative portion of our survey, almost all of  the DHS respondents (14 out of 16) who replied to a question on  timeliness stated that time constraints during the consultation process  either led to disruptions or may have hindered their ability to properly  identify and consult with appropriate subject matter experts within their  agency. One respondent noted that accelerated timeframes negatively  impacted their ability to ensure substantive engagement with the state,  local, and tribal leaders while another noted that “arbitrary” and  “unpredictable” timeframes resulted in time away from their mission. In  addition, in our interviews with federal stakeholders, officials from five of  nine federal agencies stated that involving the agencies earlier in the  process could help alleviate compressed timeframes in the final review  phase.
According to program management standards, stakeholder and program  time management are recognized practices, among others, for operating  programs successfully. Because stakeholders are defined as those  whose interests might be affected by the program outcomes and play a  critical role in the success of any program, stakeholder consultations  should include an active exchange of accurate, consistent and timely  information that reaches all relevant stakeholders. Time management is  necessary for program components and entities to keep the overall  program on track and produce a final product. According to SPAR  officials, the time constraints for the 2014 QHSR were related to a  combination of factors such as a communication delay associated with  the government shutdown and the change in DHS leadership in 2013,  delays in the report vetting process, and inadequate planning in the early  stages of the review. For example, the delay in the release of the terms  of reference document led to delays in other areas such as the  establishment of study groups and ultimately delayed the efforts to  conduct outreach to stakeholders. DHS officials responsible for  conducting stakeholder outreach expressed frustration with the delays  and noted that they were unable to move forward with planned outreach  activities. In addition, DHS SPAR officials acknowledged that engaging  the stakeholder community early and regularly needs to be built into the  planning for the review.
We continue to believe that taking steps to ensure stakeholders have the  time necessary to provide quality input is critical for DHS to incorporate  their expertise and perspectives. We recognize the possibility of  leadership changes and report vetting delays, but it is precisely because  of such contingencies that planning for and building in sufficient time  would better ensure that all key stakeholders can fully participate in the  event of such delays. As a result of time constraints, the QHSR process  may not have benefitted from the full participation of all relevant  stakeholders and may have missed opportunities to incorporate their  perspectives.
section_title: Improved Use of Collaboration Mechanisms Could Benefit Stakeholder Consultations
subsections: section_title: DHS Did Not Provide Sufficient Feedback Opportunities During Meetings
paragraphs: According to QHSR planning documents and DHS Policy officials,  obtaining and incorporating the perspectives of the entire homeland  security enterprise, including relevant federal and nonfederal  representatives was a key step in the development of the 2014 QHSR.  Federal and nonfederal survey respondents indicated, however, that  QHSR stakeholder outreach meetings did not allow for interactive  exchange between stakeholders and DHS officials. In the narrative  responses to our survey, 43 of 61 respondents to one question in our  survey stated that collaboration with stakeholders could be improved.  These respondents included federal and nonfederal stakeholders. For  example, one survey respondent noted that stakeholders were asked to  react to information provided by DHS rather than participating in  formulating the approach and execution of the studies.
In addition to indications of collaboration challenges in our stakeholder  survey results, during our interviews with federal departments and  agencies, six of nine federal representatives stated that they felt DHS’s  communication with them was “one-way” and led to an overall sense that  stakeholder input was not valued or genuinely sought after.
DHS also held working meetings with internal DHS QHSR stakeholders,  including senior and component leadership, to elicit input and feedback.  However, the After Action Report also stated that some stakeholders  believed that meetings included issues that had already been resolved or  decided upon by DHS leadership. Specifically, the report noted that the  DHS components viewed the meetings as opportunities to “check the  box” that internal DHS stakeholders were consulted instead of true  opportunities for input. The report recommended that future QHSR  stakeholder consultation sessions follow a structure where leaders ask for  information, groups present findings for discussion, and leaders review  and provide feedback until a resolution is reached.
Project management standards state that effective stakeholder  engagement involves gaining and maintaining stakeholder “buy in” for the  program’s objectives, benefits, and outcomes throughout the effort.  Specifically, program managers should employ communication methods  that target specific stakeholder needs, expectations, and wants and also  consider interactive “two way” communication since it is the most efficient  way to ensure common understanding by all participants on specified  topics.
DHS SPAR officials stated that although they initially planned to employ  interactive meetings, particularly to reach the nonfederal stakeholders,  staff, resource, and time constraints restricted DHS’s ability to engage in  a fully collaborative effort and hindered its ability to employ other  mechanisms that would have enabled more interactive feedback. DHS  used mechanisms such as existing groups within DHS and events to  connect with nonfederal stakeholders, but DHS officials stated that these  events did not provide interactive feedback with the audience. For  example, DHS briefed stakeholders from the private sector via meetings  and although these meetings are helpful in contacting relevant  stakeholders and informing them of QHSR issues, these sessions did not  allow for stakeholder responses and exchange. DHS SPAR officials  further acknowledged that the effort to reach out to stakeholders via  electronic tools became more of an information sharing exercise instead  of a means for interactive feedback. In addition, DHS’s 2014 QHSR After  Action Report noted that the online tools were used to validate study  findings instead of informing them as originally planned.
By not fostering interactive communication with federal and nonfederal  stakeholders, DHS may have missed opportunities to fully engage the  entire homeland security enterprise and thereby may not have fully  informed the QHSR effort. A means for engaging both federal and  nonfederal stakeholders in interactive communication could help ensure  common understanding across the enterprise and elicit more robust  contributions for stakeholders.
section_title: DHS Did Not Sufficiently Clarify Roles and Responsibilities for Detailees
paragraphs: In order to facilitate collaboration and information sharing among the  internal DHS stakeholders, DHS Policy requested and received detailee  staff at the supervisory level from each of the DHS internal components to  serve a 6 month assignment with the QHSR core team led by DHS  SPAR. Specifically, detailees were to serve as primary analysts  supporting the QHSR studies and perform liaison activities between DHS  SPAR and component leadership. According to DHS SPAR officials, the  detailee program was a useful mechanism in ensuring that DHS  components had a voice in the QHSR process. DHS SPAR officials  added that the detailees provided valuable insights and contributions to  the QHSR process. However, in the narrative portion of our survey and in  our interviews with DHS components, respondents expressed concerns  regarding the purpose and utilization of these staff. For example, DHS  officials at 4 of 9 components told us that the staff detailed to support the  QHSR effort either could have used additional clarity or did not have a  clear understanding of their roles and responsibilities.  One of the  detailees stated that they had not been used to communicate information  between the component and DHS SPAR. In addition, the DHS After  Action Report acknowledged that some detailees were not utilized  effectively, seemed to lack clear tasks to perform on a regular basis, and  did not have subject matter expertise needed to support the studies. For  example, the report noted that some detailees struggled comprehending  the terminology used during the studies and therefore were confused as  to how they could best support the effort. The After Action report stated  that the components receive more advanced notice on the type of  detailee that would be needed to support the QHSR process and consider  identifying the study topics in advance of requesting the detailees so that  the components can provide detailees with relevant expertise.
According to key principles of inter-agency collaboration, identifying and  clarifying the roles and responsibilities of staff, such as component  detailees, by ensuring that they have appropriate knowledge, skills, and  abilities, can strengthen collaboration. Such clarification would better  enable DHS to derive the benefit of obtaining homeland security  enterprise-wide input and feedback. However, DHS SPAR officials stated  that some of the confusion about detailee roles and responsibilities may  have been because SPAR received a detailee at the supervisory level  from each component, but later recognized that it needed individuals with  relevant analytical skill sets to inform the QHSR—not just supervisory  skills. Although the initial DHS Policy letter requesting detailees from  components specified the need for analytical skills, it did not specify the  types of or areas of expertise needed. The After Action Report also stated  that by requesting study topics earlier there would be additional time to  identify detailees with relevant subject matter expertise. SPAR officials  stated that in the future, they would consider specifying the types of  expertise needed from the component staff in advance of their request.  Without recruiting detailees with the appropriate subject matter expertise  and identifying and clarifying the roles and responsibilities, component  stakeholders may not have the information that they need to fully  participate and be invested in the process, and may thereby not be fully  committed to the QHSR effort.
section_title: Conclusions
paragraphs: The homeland security threats to the nation are continuously evolving and  in recent years have included cyber security breaches, mass shootings,  and natural disasters that have devastated portions of the United States.  In order to effectively respond to the nation’s changing security  conditions, DHS must regularly review and refine the nation’s homeland  security strategy, missions, and goals. Both the first and second  Quadrennial Homeland Security Reviews were massive undertakings  involving considerable time and resources on the part of DHS, as well as  the input of numerous federal agencies, state and local government  entities, and academics. In developing its second QHSR, DHS took  important steps toward assessing homeland security risks and also  sought to engage a wider audience of stakeholders in developing and  reviewing the strategy. DHS officials said the QHSR risk assessment was  highly complex and they faced limited time, personnel, and available data  with which to conduct and document the assessment. However, not fully  documenting the assessment and its methodology lessens the  defensibility of the results and limits the assessment’s reproducibility for  future QHSRs. Furthermore, these data and methodological limitations,  along with concerns on the part of senior leadership with using risk  assessment results to support a relative risk-ranking of the department’s  strategic priorities, resulted in DHS missing an opportunity to use the  QHSR to more efficiently implement programs, strategies, and policies to  address different levels and types of risks. Last, insufficient forums for  multi-directional collaboration and DHS SPAR’s unclear expectations of  component detailees resulted in DHS missing opportunities to engage  with the full range of homeland security stakeholders capable of informing  the QHSR effort. For its next QHSR, DHS has an opportunity to build on  its progress to date in order to ensure its missions, goals, and strategy  addresses the most significant homeland security risks and is  representative of the full breadth of homeland security stakeholders.
section_title: Recommendations for Executive Action
paragraphs: To ensure the quality of the risk assessments used to inform its future  QHSR processes, the Secretary of Homeland Security should direct the  Assistant Secretary for Policy to ensure future QHSR risk assessment  methodologies reflect key elements of successful risk assessment  methodologies, such as being:
Documented, which includes documenting how risk information was  integrated to arrive at the assessment results,
Reproducible, which includes producing comparable, repeatable
Defensible, which includes communicating any implications of  uncertainty to users of the risk results.
To enable the use of risk information in supporting resource allocation  decisions, guiding investments, and highlighting the measures that offer  the greatest return on investment, the Secretary of Homeland Security  should direct the Assistant Secretary for Policy to refine its risk  assessment methodology so that in future QHSRs it can compare and  prioritize homeland security risks and risk mitigation strategies.
To ensure proper management of the QHSR stakeholder consultation  process, the Secretary of Homeland Security should direct the Assistant  Secretary for Policy to identify and implement stakeholder meeting  processes to ensure that communication is interactive when project  planning for the next QHSR.
To ensure proper management of the internal QHSR stakeholder  consultation process, the Secretary of Homeland Security should direct  the Assistant Secretary for Policy to clarify component detailee roles and  responsibilities when project planning for the next QHSR.
section_title: Agency Comments and Our Evaluation
paragraphs: We requested comments on a draft of this report from DHS. On April 4,  2016, DHS provided written comments, which are reprinted in appendix  V. DHS concurred with our four recommendations and described actions  planned to address them. With regard to our first recommendation that  DHS ensure future QHSR risk assessment methodologies reflect key  elements of successful risk assessment methodologies—such as being  documented, reproducible, and defensible—DHS stated that it believed  that the 2013 Homeland Security National Risk Characterization  (HSNRC) was complete, documented, reproducible, and defensible,  consistent with the National Infrastructure Protection Plan guidance. DHS  further stated that the HSNRC was developed using sound risk analysis  methodologies informed by well-established analytical protocols and  reproducible data provided by multiple DHS and component programs  and subject matter experts, and that analysis was reviewed by the DHS  Risk Executive Steering Committee and documented throughout the  development of the HSNRC.
However, our review of DHS’s HSNRC documentation, as well as  documentation on the Homeland Security Strategic Environment  Assessment (of which the HSNRC is a part), found that DHS did not fully  document how its risk results (from both its quantitative and qualitative  analyses) were synthesized to generate its list of standout hazards and  included only a limited description of data DHS relied on to generate its  results. Further, DHS did not provide a description of how any data or  methodological limitations should be interpreted by users of the results.  Our review of DHS’s documentation also found that DHS did not provide  an explanation of how all of the various Homeland Security Strategic  Environment Assessment elements—the system maps, the risk ranking  from the HSNRC, the trends analysis, or the threat reports—were  combined to generate the QHSR risk narrative or other risk findings. DHS  provided an explanation that its various analyses were combined in order  to generate insights, but did not provide descriptions of the weighting  factors, assumptions, and subjective judgments that went into  synthesizing what DHS referred to as static and dynamic risk. In addition,  beyond informing a strategic context, it is unclear how (if at all) decision- makers should interpret and use the QHSR risk results. Incorporating the  key principles of a successful risk assessment methodology can help  DHS demonstrate the basis of its risk management strategies and help  ensure their validity for guiding QHSR mission and goals.
Regarding our second recommendation that DHS refine its risk  assessment methodology so that in future QHSRs it can compare and  prioritize homeland security risks and risk mitigation strategies, DHS  stated that it will continue to refine its risk assessment methodology and  expand its use of comparative risk analysis in the next HSNRC.  Regarding our third recommendation that DHS identify and implement  stakeholder meeting processes to ensure that communication is  interactive when project planning for the next QHSR, DHS stated it is  developing an extensive stakeholder outreach plan to include leveraging  the Homeland Security Information Network and its access to a broad  community of federal, state, and local stakeholders across the homeland  security enterprise. The outreach plan is to include using on-line tools, in- person subject matter expert interviews, and extensive interagency  coordination to facilitate information exchanges, document reviews, and  feedback throughout the analysis and review phases of the next QHSR.  In regards to the fourth recommendation that DHS clarify component  detailee roles and responsibilities when project planning for the next  QHSR, DHS stated that it will request DHS directorate and component  detailees with specific skill sets and provide clearly defined timelines and  objectives to support the QHSR.  DHS also provided technical comments,  which we incorporated as appropriate.
We also requested comments on a draft of this report from the  Departments of Agriculture, Defense, Health and Human Services,  Justice, State, Transportation, Treasury, and Veterans Affairs, the  Environmental Protection Agency, and the Office of the Director of  National Intelligence. The Department of State provided comments in an  email received March 31, 2016, emphasizing the importance of risk- based resource allocation as well as the importance of linking operations  and performance metrics to risk information. In an e-mail received from a  departmental liaison on March 23, 2016, the Department of Agriculture  indicated that it had no comments on the report. In e-mails received from  departmental liaisons on March 29, 2016, the Department of  Transportation and Department of Veterans Affairs indicated that they  had no comments on the report. In e-mails received from departmental  liaisons on March 30, 2016, the Department of Health and Human  Services, Department of Defense, and the Office of the Director of  National Intelligence indicated that they had no comments on the report.  In an e-mail from a departmental liaison on April 1, 2016, the Department  of Justice indicated that it had no comments on the report. In an e-mail  received on April 4, 2016 from an agency liaison, the Environmental  Protection Agency indicated that it had no comments on the report. In an  e-mail received on April 8, 2016, from the Department of the Treasury’s  Director for Emergency Programs the department indicated that it had no  comments on the report.
We are sending copies of this report to the Secretaries of Agriculture,  Defense, Health and Human Services, Homeland Security, State,  Transportation, Treasury and Veterans Affairs; the Attorney General; the  Director of National Intelligence; Administrator of the Environmental  Protection Agency; and selected congressional committees. In addition,  this report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (213) 830-1011 or vonaha@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix V.
section_title: Appendix I: 9/11 Commission Act Quadrennial Homeland Security Review (QHSR) Reporting Elements
paragraphs: Of the nine reporting elements specified in the Implementing  Recommendations of 9/11 Commission Act of 2007 (9/11 Commission  Act) for inclusion in the QHSR report, the Department of Homeland  Security (DHS) addressed five and did not fully address four through the  2014 QHSR report, finalized in June 2014, and the DHS Fiscal Years  2014-2018 Strategic Plan (Strategic Plan), released in December 2014,  as shown in table 4. Elements DHS addressed in those documents  included a description of homeland security threats, discussions of the  status of cooperation among federal agencies and between the federal  government and state, local, and tribal governments, an explanation of  underlying assumptions for the QHSR report, and other matters  considered important. Elements not fully addressed included a prioritized  list of homeland security missions, a description of the budget plan  required to execute the full range of missions, and an assessment of the  alignment of DHS with the QHSR missions.
DHS officials agreed with our overall assessment, but asserted that the  specifications for some of the elements we found as not fully addressed  were reported through other relevant documents. For example, according  to DHS officials, the specification for identifying a budget plan was met  through the FY2015-2019 Future Years Homeland Security Program  report, published on August 11, 2014. DHS also stated that alignment of  human resources systems was addressed in DHS’s FY2015-2019 Human  Capital Strategic Plan. For the purposes of evaluating whether DHS  addressed the QHSR reporting elements, we limited our review to the  QHSR report and DHS’s Strategic Plan, which references the QHSR  report and further articulates homeland security missions, goals and  strategies. We did not review any additional documents that DHS had  published pertaining to issues identified for inclusion in the QHSR report,  like the budget or human resources-related matters. Therefore, we did not  evaluate whether DHS addressed the reporting requirements through  means outside of these two documents.
The statute establishing the QHSR provides that each such review “shall  be a comprehensive examination of the homeland security strategy of the  Nation, including recommendations regarding the long-term strategy and  priorities of the Nation for homeland security and guidance on the  programs, assets, capabilities, budget, policies, and authorities of the  Department.” The statute further requires that DHS submit to Congress a  report regarding the QHSR, and identifies specific elements the report is  to include. Our review of the QHSR report and the Strategic Plan  concluded that while DHS had addressed five of the elements mandated  for inclusion, it did not fully address four of the elements. Consequently,  to the extent Congress anticipated the submission of a single  comprehensive QHSR report that fully addressed each of the nine  elements specified in the 9/11 Commission Act, rather than multiple DHS  products that may collectively address each element, the report submitted  to Congress on June 18, 2014, falls short. In addition, DHS issued the  QHSR report after December 31, 2013, the date contemplated in the 9/11  Commission Act for DHS to report on the results of the quadrennial  review conducted in 2013. According to DHS officials, DHS released the  QHSR report after this date because leadership changes, specifically the  appointment of a new Homeland Security Secretary in October 2013,  occurred prior to the completion of the 2014 QHSR and thus extended the  report’s review time frames.
section_title: Appendix II: Scope and Methodology
paragraphs: To determine the extent to which the Department of Homeland Security  (DHS) has examined and used risk information to inform the Quadrennial  Homeland Security Review and its implementation, we analyzed DHS  documentation on its risk analysis process and results for the Homeland  Security Strategic Environment Assessment (HSSEA)—the risk analysis  conducted for the QHSR. We reviewed DHS’s Current and Future  Strategic environment reports, the Systems Mapping Initiative report, the  Homeland Security National Risk Characterization (HSNRC), and  classified HSSEA results. We also reviewed the 2012 and 2013  Worldwide Threat Assessment of the U.S. Intelligence Community,  prepared by the Office of the Director of National Intelligence (ODNI),  which DHS reported relying on when assessing threat information for the  HSSEA. In addition to the HSSEA reports themselves, we analyzed all  available supporting documentation provided by DHS on the development  of these reports, including briefings, videos, meeting summaries, and  descriptions of HSSEA processes and results. We evaluated the  sufficiency of this documentation and the overall completeness of the  HSSEA risk assessment using the standards set forth in the 2013  National Infrastructure Protection Plan (NIPP) and its Supplemental Tool  on Executing a Critical Infrastructure Risk Management Approach. The  NIPP states that in order to be complete, a risk assessment must  consider threat, vulnerabilities, and consequences; must be sufficiently  documented; must be reproducible; and must be defensible. In order to  be sufficiently documented, the methodology and the assessment must  clearly document what information is used and how it is synthesized to  generate a risk estimate. Any assumptions, weighting factors, and  subjective judgments need to be transparent to the user of the  methodology, its audience, and others who are expected to use the  results. The types of decisions that the risk assessment is designed to  support and the timeframe of the assessment (e.g., current conditions  versus future operations) should be given. We also looked to a 2010  review by the National Academies of Science on DHS’s Risk  Management Approach, as well as our previous work evaluating DHS’s  approach to risk assessment, in order to ensure that we used a consistent  approach to evaluating DHS’s assessment efforts. To supplement and  clarify DHS’s written documentation on the HSSEA, we interviewed  current and former DHS officials from the Office of Strategy, Plans,  Analysis, and Risk (SPAR) who were responsible for developing the  QHSR risk analyses. We asked officials to discuss the HSSEA process  and how results were translated into the QHSR, including the extent to  which uncertainty in the risk estimates was incorporated into the final  results and communicated to users of the results—a NIPP aspect of a  defensible risk assessment. Further, we evaluated DHS’s use of risk  information in the QHSR to risk management guidance in the NIPP and  DHS’s Risk Management Fundamentals, as well as our prior work on key  characteristics for risk assessment and management. Last, we met with  the authors of a December 2015 paper published in Homeland Security  Affairs on a comparative ranking of homeland security hazards in order to  discuss current challenges and developments in the field of homeland  security risk analysis.
To determine the extent to which DHS has aligned its budget and  performance measures with the mission goals in the QHSR, we analyzed  DHS documents related to the 2014 QHSR and the fiscal years 2014- 2018 Strategic Plan, including: DHS Budget in Brief documents from 2015  through 2017 to determine DHS’s budget priorities since issuance of the  first QHSR; fiscal years 2012 through 2015 Future Years Homeland  Security Program (FYHSP) reports; excerpts from DHS’s fiscal years  2017-2021 and 2018-2022 resource planning guidance to determine the  extent to which budget guidance reflects the DHS missions and goals;  and DHS reports on its progress implementing a new common  appropriations structure. Further, we reviewed Congressional Research  Services reports on DHS’s appropriations from fiscal year 2015 and 2016.  We also interviewed DHS officials from the Office of the Chief Financial  Officer, Office of Strategy, Plans, Analysis, and Risk (SPAR), and from  U.S. Customs and Border Protection (CBP), U.S. Coast Guard (USCG),  Federal Emergency Management Agency (FEMA), U.S. Immigration and  Customs Enforcement (ICE), National Protection and Programs  Directorate (NPPD), and Transportation Security Administration (TSA).  We selected these components based on their share of the overall DHS  fiscal year 2015 budget, breadth of homeland security responsibilities,  and other factors. Views from these components are not generalizable to  all DHS components. We interviewed component officials to determine  the extent to which they used DHS guidance in developing their annual  budget requests and the extent to which such guidance reflects the  QHSR missions and goals. With respect to performance measures we  reviewed DHS’s Annual Performance Plans from fiscal years 2013-2015  and 2014-2016, reviewed our prior work on key attributes of successful  performance measures, reviewed the GPRA Modernization Act of 2010  (GPRAMA), and interviewed DHS officials from SPAR and the office of  Program Analysis and Evaluation (PA&E) and from CBP, USCG, FEMA,  ICE, NPPD, and TSA components in order to determine the extent to  which DHS developed performance measures that aligned with mission  goals. We compared DHS’s process for monitoring and measuring  performance to internal control standards and our prior work on best  practices for implementation of strategies and initiatives.
To determine the extent to which DHS consulted with stakeholders in  developing the QHSR, we distributed a web-based survey to 222 QHSR  stakeholders identified by DHS based on 2014 QHSR participation. The  stakeholders included representatives from 9 federal departments and  agencies; 12 DHS components, directorates and offices; about 20 state,  local, tribal, private and academic organizations. The stakeholders DHS  identified did not include those who participated solely online and, as a  result, we did not include these individuals within the scope of our review.  We conducted 5 pretests with respondents from DHS, other federal  agencies, and related think tanks to verify that (1) the questions were  clear and unambiguous, (2) terminology was used correctly, (3) the  questionnaire did not place an undue burden on respondents, (4) the  information could feasibly be obtained, and (5) the survey was  comprehensive and unbiased. We made changes to the content and  format of the questionnaire after the pretests based on the feedback we  received. Before we administered the survey, we revised the  questionnaire to reflect comments from an independent reviewer within  GAO.
We identified 40 respondents that either had an undeliverable email  address, were no longer employed with the federal government and/or did  not participate in the 2014 QHSR. We removed these stakeholders from  our final distribution list, resulting in about 180 possible respondents out  of the original 222 that were identified by DHS. Questionnaires were  completed by 93 individual stakeholders overall, resulting in a response  rate of fifty-one percent.
Since the survey is not based on a sample, it does not have sampling  errors. However, the practical difficulties of conducting any survey may  introduce errors, commonly referred to as non-sampling errors. For  example, difficulties in interpreting a particular question, sources of  information available to respondents, or entering data into a database or  analyzing them can introduce unwanted variability into the survey results.  We took steps in developing the questionnaire, collecting the data, and  analyzing them to minimize such non-sampling error. For example, social  science survey specialists designed the questionnaire in collaboration  with GAO staff with subject matter expertise. Then, we pre tested the  draft questionnaire to ensure that the questions were relevant, clearly  stated, and easy to understand. When we analyzed the data, an  independent analyst checked all computer programs. Since this was a  web-based survey, respondents entered their answers directly into the  electronic questionnaire, eliminating the need to key data into a database  and thereby minimizing error.
We also asked open-ended questions regarding the QHSR stakeholder  consultation process, including ways in which DHS elicited stakeholder  input, how revisions to timeframes may have impacted stakeholder  responsibilities, any challenges in collaborating with DHS and  suggestions for improving future QHSRs. Of the 93 individual  stakeholders that responded to our survey, 74 respondents provided  narrative comments. We analyzed these comments to determine common  benefits and challenges they identified regarding DHS consultations  during the QHSR. The comments received from these respondents are  not generalizable to the entire group of stakeholders, but the feedback  provided insights into stakeholder perspectives on how QHSR  stakeholder consultations were conducted and how they could be  improved. We compared DHS’s stakeholder outreach efforts to project  management standards.
To assess the extent to which the 2014 QHSR and fiscal years 2014- 2018 Strategic Plan reports addressed reporting elements listed in the  9/11 Commission Act, we determined the extent to which each element  was addressed in the reports. To accomplish this determination, three  GAO analysts independently compared the QHSR and strategic plan to  each of the nine reporting elements to determine whether each element  was addressed, not fully addressed, or not addressed. In cases when the  analysts disagreed, they reviewed and discussed their independent  assessments to reach concurrence. We considered an element  addressed if all portions of it were included in either the QHSR or  strategic plan, not fully addressed if one or more but not all portions of the  element were included, and not addressed if neither the QHSR nor the  strategic plan addressed any part of the element. In addition, we  interviewed DHS officials involved in the quadrennial review to determine  DHS’s position on how the 9/11 Commission Act reporting requirements  were implemented.
We conducted this performance audit from January 2015 to April 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix III: Hazard Scope for the DHS Homeland Security National Risk Characterization
subsections: section_title: Terrorism/Intentional Acts Aircraft as a Weapon
section_title: Chemical / Biological Food Contamination Terrorism Attack
section_title: Appendix IV: Strategies Linked to Quadrennial Homeland Security Review Missions and Goals
subsections: section_title: The following table outlines the list of strategies linked to Quadrennial Homeland Security Review missions and goals depicted in figure 5.
subsections: section_title: Mission 1: PREVENT TERRORISM AND ENHANCE SECURITY Goal 1.1: Prevent Terrorist Attacks
paragraphs: Analyze, fuse, and disseminate terrorism  information;
Deter and disrupt operations;  Strengthen transportation security; and
Counter violent extremism.
Goal 1.3: Reduce Risk to the Nation’s Critical Infrastructure, Key Leadership,  and Events  chemical, biological, radiological, and nuclear  materials and weapons.  Enhance security for the nation’s critical  infrastructure from terrorism and criminal  activity; and  Protect key leaders, facilities, and national  special security events.
Goal 2.3: Disrupt and Dismantle Transnational Criminal Organizations and  Other Illicit Actors  promote U.S. economic security and  competitiveness.  Identify, investigate, disrupt, and dismantle  transnational criminal organizations; and
Disrupt illicit actors, activities, and pathways.
Promote lawful immigration;  Effectively administer the immigration services  system; and  Promote the integration of lawful immigrants  into American society.
Prevent unlawful entry, strengthen enforcement,  and reduce drivers of unlawful immigration; and  Arrest, detain, and remove priority individuals,  including public safety, national security, and  border security threats.
Enhance the exchange of information and  intelligence on risks to critical infrastructure and  develop real-time situational awareness  capabilities that ensure machine and human  interpretation and visualization;  Partner with critical infrastructure owners and  operators to ensure the delivery of essential  services and functions;  Identify and understand interdependencies and  cascading impacts among critical infrastructure  systems;
Collaborate with agencies and the private sector  to identify and develop effective cybersecurity  policies and best practices; and
Reduce vulnerabilities and promote resilient  critical infrastructure design.
Goal 4.2: Secure the Federal Civilian Government Information Technology  Enterprise
Coordinate government purchasing of cyber  technology to enhance cost-effectiveness;  Equip civilian government networks with  innovative cybersecurity tools and protections;  and  Ensure government-wide policies and standards  are consistently and effectively implemented  and measured.
Goal 4.4: Strengthen the Cyber Ecosystem
Deter, disrupt, and investigate cybercrime.
Drive innovative and cost effective security  products, services, and solutions throughout the  cyber ecosystem;
Conduct and transition research and  development, enabling trustworthy cyber  infrastructure;
Develop skilled cybersecurity professionals;  Enhance public awareness and promote  cybersecurity best practices; and  Advance international engagement to promote  capacity building, international standards, and  cooperation.
section_title: Mission 5: STRENGTHEN NATIONAL PREPAREDNESS AND RESILIENCE Goal 5.1: Enhance National Preparedness
section_title: Goal 5.2: Mitigate Hazards And Vulnerabilities
paragraphs: Goal 5.3: Ensure Effective Emergency Response  regulation, resilient design, effective mitigation,  and disaster risk reduction measures; and  Prevent incidents by establishing, and ensuring  compliance with, standards and regulations.  Provide timely and accurate information;
Conduct effective, unified incident response  operations;  Provide timely and appropriate disaster  assistance; and  Ensure effective emergency communications.  Ensure continuity and restoration of essential  services and functions; and  Support and enable communities to rebuild  stronger, smarter, and safer.
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Ben Atwater (Assistant Director),  Nanette Barton, Chuck Bausell, Jr., Charlotte Gamble, Susan Hsu,  Thomas Lombardi, Amanda Miller, and Katrina Taylor made key  contributions to this report.