Quadrennial Homeland Security Review: Improved Risk Analysis and Stakeholder Consultations Could Enhance Future Reviews
Released: 2016-04-15T00:00:00

Letter

Background

The 9/11 Commission Act Establishes QHSR Reporting Elements

Section 707 of the Homeland Security Act of 2002, as amended by the 9/11 Commission Act, requires a review of the nation’s homeland security every four years. Under the 9/11 Commission Act, DHS is responsible for developing the QHSR and for submitting a report that is to include multiple elements, such as: a description of the threats to the assumed or defined national homeland security interests of the United States; the national homeland security strategy, including a prioritized list of the critical homeland security missions of the United States; an assessment of the organizational alignment of DHS with the applicable national homeland security strategy and the homeland security mission areas outlined; and a discussion of the status of cooperation among federal agencies in the effort to promote national homeland security, among other elements.

DHS’s 2014 QHSR Process

The Department initiated the second QHSR in summer 2012. Led by SPAR within the Office of Policy, the department reviewed the homeland security strategic environment as well as the DHS missions and goals. In 2013, DHS issued its terms of reference outlining the framework for conducting the second quadrennial review and identified threats and assumptions to be used in conducting the review. The QHSR was conducted in four phases, as shown in figure 1, and released in June of 2014.

Based on the assessments conducted in the preparation phase, DHS leadership directed five QHSR studies, as shown above. Each study group was composed of officials from across DHS offices and components, led by a DHS SPAR official, and facilitated by an independent subject matter expert from the Homeland Security Studies and Analysis Institute. These study groups conducted their analysis over a 6 month period and shared their work products, such as outline missions and assumptions, with other stakeholder groups in order to develop goals and objectives for each homeland security mission. At the end of the study group period, DHS senior leadership, including the Deputy Secretary of Homeland Security, the Office of General Counsel, and office and component heads, met multiple times to review and discuss the study group recommendations. DHS SPAR consolidated the study groups’ recommendations into a draft QHSR report and obtained and incorporated feedback on the draft report from other federal agencies and stakeholder groups, including stakeholders listed in the 9/11 Commission Act with which DHS was to consult in conducting the 2014 QHSR review.

DHS solicited input from various stakeholder groups in conducting the second QHSR. The 9/11 Commission Act specifically required DHS to consult with the heads of seven federal agencies. DHS consulted with these agencies and also sought input from a range of other stakeholders, including its directorates, offices, and components; other federal departments and agencies; and nonfederal governmental and nongovernmental entities and representatives, such as state, local and tribal governmental associations, members of Congress, private sector representatives, academics, and other policy experts. DHS SPAR consulted with the Office of Intergovernmental Affairs and the Office of State and Local Law Enforcement to identify and contact relevant stakeholder organizations.

DHS Risk Management Framework

We, Congress, the 9/11 Commission, and others have recommended that federal agencies with homeland security responsibilities use a risk management approach to help ensure that finite resources are dedicated to assets or activities considered to have the highest security priority. In its Risk Management Fundamentals and other guidance, DHS articulates a risk management framework to guide its homeland security planning, preparation, and execution of its missions. DHS states that risk management is the process for identifying, analyzing, and communicating risk and accepting, avoiding, transferring, or controlling it to an acceptable level. The National Infrastructure Protection Plan (NIPP) also defines DHS’s risk management framework and includes key principles for an effective risk assessment. Specifically, a risk assessment should be complete, documented, reproducible, and defensible, as defined in table 1.

DHS Budget Formulation Process

DHS uses a planning, programming, budgeting, and execution (PPBE) process to allocate resources. DHS’s PPBE process produces the multi- year funding plans presented in the Future Years Homeland Security Program (FYHSP). According to DHS guidance, at the outset of the annual PPBE process, the department’s Office of Policy and the Chief Financial Officer (OCFO) should provide planning and fiscal guidance, respectively, to the department’s 16 component agencies. In accordance with this planning and fiscal guidance, the components should produce 5- year funding plans—called Resource Allocation Plans—that are submitted to the CFO and reviewed by DHS’s senior leaders. DHS senior leadership is expected to modify the plans in accordance with their priorities and assessments and submit them to the Office of Management and Budget (OMB), which uses the plans to inform the President’s annual budget request. DHS guidance establishes approximate timelines for when guidance is provided to components and when budget plans are due, as shown below in figure 3.

DHS Mission Goals and Performance Measures

The 2014 QHSR outlines the same five missions set forth in the first QHSR in 2010, but the 2014 review refined the missions to reflect evolving threats. The five missions are to (1) Prevent Terrorism and Enhance Security, (2) Secure and Manage Our Borders, (3) Enforce and Administer Our Immigration Laws, (4) Safeguard and Secure Cyberspace, and (5) Strengthen National Preparedness and Resilience. Changes to the DHS missions were primarily in Mission 4 to Safeguard and Secure Cyberspace. Specifically, the 2014 QHSR provides enhanced goals for cybersecurity protection that include leveraging technology and enhancing investigative capabilities, as shown in figure 4.

DHS assesses and reports on progress for the 2014 QHSR mission goals in its Annual Performance Report, which provides suites of performance measures that assess progress for each QHSR mission goal area. The Annual Performance Report includes performance targets as well as information on agency priority goals, initiatives, and challenges. The performance results identified in the report are organized around the missions and goals outlined in the DHS Strategic Plan.

The DHS Fiscal Year 2014-2018 Strategic Plan focuses on how DHS plans to implement the goals laid out in the 2014 QHSR. Specifically, the Strategic Plan describes the QHSR homeland security missions and goals and lists strategies to achieve the goals. For example, for Mission 1 (Prevent Terrorism and Enhance Security), to achieve Goal 1.1 (prevent terrorist attacks), one strategy listed in the Strategic Plan is to analyze, fuse, and disseminate terrorism information by sharing it with stakeholders across the homeland security enterprise. In addition, for Mission 4 (Safeguard and Secure Cyberspace), to achieve Goal 4.2 (secure the federal civilian government information technology enterprise), one strategy listed in the Strategic Plan is to ensure government-wide policies and standards are consistently and effectively implemented. See figure 5 for a list of additional strategies linked to homeland security missions and goals.

Move mouse over goal to see the objectives. For a noninteractive version, see appendix IV.

Better Documentation of the QHSR Risk Assessment Process and Prioritization of Results Could Strengthen DHS Risk Management Efforts

DHS Sought to Identify Strategically Significant Risks for the 2014 QHSR

DHS conducted a risk assessment for the second QHSR, which it referred to as the Homeland Security Strategic Environment Assessment (HSSEA). According to DHS, the purpose of the HSSEA was to characterize those risks, threats, current and future trends, and critical uncertainties that will most affect homeland security in the 2015 to 2019 timeframe. The HSSEA is the product of several different assessments that DHS developed for this effort, including the Current Strategic Environment, Future Strategic Environment, Homeland Security Planning Threat Assessment, Homeland Security National Risk Characterization (HSNRC), and the System Mapping Initiative. All combined, DHS intended these documents to inform the QHSR’s narrative describing the homeland security strategic environment as well as informing the selection of priority strategic issues to study during the 2014 QHSR, among other goals as shown in figure 6.

DHS Did Not Incorporate All Elements of a Successful Risk Assessment in the 2014 QHSR

For the HSSEA, DHS evaluated threat information and assessed relative likelihood and consequences across various hazards that DHS determined to be significant, including intentional acts of terrorism, natural hazards, and technological accidents and infrastructure failures; however, DHS did not incorporate other key elements of a successful risk assessment into its risk assessment methodology. Specifically, DHS did not clearly document how its various analyses were synthesized to generate risk results; did not design an assessment that could produce comparable, repeatable results; and did not communicate the implications of any uncertainty in the results. DHS’s performance in each NIPP element—complete, documented, reproducible, defensible—is discussed in further detail below.

Complete

The HSSEA generally follows the NIPP standard for a complete risk assessment in that it assesses consequence, vulnerability, and threat. Regarding threat information, DHS officials said that they relied on two primary sources—the classified Homeland Security Planning Threat Assessment, developed by DHS Intelligence and Analysis (I&A), and the publically available 2012 Worldwide Threat Assessment of the US Intelligence Community, which is prepared by the Office of the Director of National Intelligence (ODNI). The ODNI assessment discusses the rapid changes in the threat environment which ODNI states is now more diverse and interconnected than any other time in history. DHS officials said that they used the unclassified assessment as well as select assessments from the I&A Homeland Security Planning Threat Assessment so that they did not face classification challenges when sharing information with partners. DHS I&A reported using I&A and intelligence community data, as well as data from component-level intelligence units. Further, the ODNI-developed assessment had been vetted through the intelligence community, which DHS officials said added to its legitimacy.

Regarding consequence and vulnerability, DHS primarily assessed these two factors in the Homeland Security National Risk Characterization (HSNRC), where it evaluated the likelihood and potential consequences of 40 hazards ranging from acts of terrorism to hurricanes. DHS compared the impacts of these hazards according to their direct economic impact, loss of life, and potential for injuries and illnesses in order to identify what the department referred to as standout risks—risks that DHS believed would result in substantial negative consequences— and to allow decision makers to evaluate risk management alternatives in terms of their effect on increasing or decreasing risk. Additionally, DHS sought to increase the usefulness of the QHSR risk assessment by including an analysis of current and future strategic trends into its broader analysis. Specifically, DHS officials said that the System Mapping Initiative, 2012 Current Strategic Environment, and 2013 Future Strategic Environment reports provided important insights on trends and other factors for the QHSR and resulted in what DHS referred to as a dynamic risk assessment.

Documented

Documented The methodology and the assessment must clearly document what information is used and how it is synthesized to generate a risk estimate. Any assumptions, weighting factors, and subjective judgments need to be transparent to the user of the methodology, its audience, and others who are expected to use the results. The types of decisions that the risk assessment is designed to support and the timeframe of the assessment (e.g., current conditions versus future operations) should be given.

As mentioned previously, the HSSEA is the product of several different assessments and was intended to be dynamic—in that, according to DHS, it captures possible future scenarios and can help decision-makers be more strategic—but DHS did not fully document how the findings and results of the various assessments were synthesized to generate the QHSR risk results, among other issues.

For the HSNRC—where DHS ranked various hazards according to their likelihood and potential consequences—DHS did not fully document how its results (from both its quantitative and qualitative analyses) were synthesized to generate its list of standout hazards—a key element of the NIPP standards for documentation. Specifically, in the documentation provided by DHS, we identified DHS’s steps for conducting the HSNRC, the hazard scope, consequence categories, and methodological caveats. In separate classified documentation, we reviewed a rank-ordering of those hazards that DHS SPAR officials said resulted from its HSNRC analysis. However, DHS’s documentation included only a limited description of how and what data DHS relied on to generate its standout hazards, and no description of how any data or methodological limitations should be interpreted by users of the results.

Additionally, DHS’s documentation included only limited information on data sources and the specific roles of subject matter experts. For example, regarding data sources, DHS cited “historical record” and “subject matter expert judgment,” without providing further detail on what record or judgements were relied upon. In classified Risk Summary Sheets, DHS officials reported documenting additional detail about each hazard, including the data sources and likelihood and consequence estimates; however, this documentation does not clarify how the information was synthesized to generate the final risk analysis results. In addition, DHS reported that in light of limited available data, it relied on the expertise of subject matter experts; however, DHS did not document how these subjective judgements were weighed against other available data to generate results, and it is unclear how much or how little the subject matter experts’ opinions were the primary basis for DHS’s findings. In a 2010 review of DHS’s approach to risk assessment, the National Academies also found that DHS had not documented how subject matter expert assumptions were made, which limits the ability to test the validity of those assumptions, bring to light dissenting views of experts, or inform future updates and improvements to the risk models.

Last, DHS did not document or provide an explanation of how all of the various HSSEA elements—the system maps, the risk ranking from the HSNRC, the trends analysis, or the threat report—were combined to generate the QHSR risk narrative and other risk findings. According to the NIPP, sufficient documentation can help improve or modify a risk methodology so that the investment and expertise they represent can be used to support risk management activities. DHS provided an explanation that its various analyses were combined in order to generate insights, but no other description of the weighting factors, assumptions, and subjective judgments that went into synthesizing static and dynamic risk in the HSSEA. DHS did, however, describe the types of decisions the HSSEA is designed to support, as called for in the NIPP. Specifically, the HSSEA should, according to DHS, reveal blind spots, place homeland security events into a broader context, and identify key challenges and opportunities, among other goals. However, beyond informing a strategic context, it is unclear how decision-makers should use and interpret the HSSEA results.

Reproducible and Defensible

Reproducible The methodology must produce comparable, repeatable results, even though assessments of different critical infrastructure and key resources may be performed by different analysts or teams of analysts. It must minimize the number and impact of subjective judgments, leaving policy and value judgments to be applied by decision makers. Defensible The risk methodology must logically integrate its components, making appropriate use of the professional disciplines relevant to the analysis, as well as be free from significant errors or omissions. Uncertainty associated with consequence estimates and confidence in the vulnerability and threat estimates should be communicated.

DHS’s lack of documentation also limits the reproducibility and defensibility of the results, since the assessment cannot easily be validated or the assumptions tested, further hindering DHS’s ability to improve future iterations of the assessment. Specifically, in the HSNRC— the element of the HSSEA that seeks to generate risk rankings—DHS evaluates the likelihood and consequences of 40 hazards that cover terrorism, natural hazards, and technological accidents or infrastructure failures. However, given the level of uncertainty cited regarding the analysis of these hazards, it is unclear how DHS determined their risk- based rank-order, as well as what findings could be drawn from the results. For example, with available data, DHS found that it could only compare consequences across three consequence categories—fatalities, injuries and illnesses, and direct economic loss. The specific definitions for injuries and illness as well as direct economic loss vary across the threats and hazards, and DHS caveated its findings by noting that some hazards used different definitions of economic consequences, which resulted in substantially different levels of consequences, thus limiting the hazards’ comparability and making it difficult to determine their relative severity. Last, DHS caveated its findings by acknowledging that there are large bands of uncertainty, especially for those estimates that rely on subject matter expert judgement. Although DHS stated these and other limitations in its supporting documentation and on the presentation of its results, DHS did not communicate any implications this uncertainty would have on use and interpretation of the results except to say that simultaneously considering multiple consequence categories makes ascertaining the top risks very difficult.

Despite these limitations, DHS did provide a clear explanation of its guiding principles for selecting the HSNRC hazard scope, which included national-level significance, relevance to DHS, relevance to the 2014 QHSR, and availability of data. Further, DHS provided definitions of all the hazards for which it evaluated frequencies and consequences, as well as which consequences it considered. DHS also explained that it evaluated hazard frequencies using the USCG’s National Maritime Security Risk Assessment frequency categories (which looks at frequencies ranging from over 550 events per year to 1 event every 180 or more years). Last, the technical working group that developed the HSNRC provided several recommendations for future HSNRC work and a list of next steps, which included developing a departmental risk modeling and analytical resource toolbox.

DHS officials said that conducting and documenting a risk assessment of this scale was a complex undertaking which was limited by both time and available personnel and that finding comparable or reliable data was challenging. Officials said that the QHSR risk assessment process was continually evolving and the assessment team was learning and adapting as they conducted the analysis. Nonetheless, in its Risk Management Fundamentals, DHS states that those affected by a risk management approach should be able to understand how results were derived and the process for doing so. Further, the NIPP provides guidance on how to document a risk assessment methodology and key information—such as the implications of uncertainty in the estimates, such as uncertainty in the number of fatalities or dollars of economic loss—that should be communicated to users of the risk results. Moreover, without sufficient documentation of its HSSEA risk methodology, DHS is hindered in its ability to reproduce the assessment for future QHSRs and limits the defensibility of the results. Incorporating the key principles of a successful risk assessment methodology can help DHS demonstrate the basis of its risk management strategies and help ensure their validity for guiding QHSR missions and goals.

QHSR Describes Various Risks Facing the Nation, but Prioritization of Risks Could Enhance DHS Risk Management Efforts

In the QHSR, DHS described a range of strategic risks—from the evolving terrorist threat to the impacts of aging critical infrastructure systems—but did not compare or prioritize risks to identify where mitigation efforts are most needed, as called for in the NIPP. Specifically, DHS describes the homeland security hazards it assessed as (1) strategic priorities, (2) the key areas of change (which DHS refers to as the threats and hazards to the nation), (3) the prevailing challenges that pose the most risk, and (4) areas of ongoing priority and emphasis, as shown in figure 7.

For the QHSR risk analysis, DHS officials said they not only wanted to develop a homeland security risk characterization, but to augment this information to provide more useful tools for DHS-wide and component decision making. However, despite DHS’s goals for the QHSR assessment, officials from the six components we met with (CBP, USCG, FEMA, ICE, NPPD, and TSA) stated that the risk results reported in the QHSR did not affect their component’s resource allocation, operational planning, or strategic planning decisions. In addition, officials from 5 of the 6 components reported not using any of the stand-alone QHSR risk products in their own risk assessment or strategic planning efforts. Component officials reported that the QHSR risk results were highly generalized and encompassed essentially all of DHS’s missions—in other words, they did not offer priorities or a sense of relative risks. Component officials added that since the QHSR results were not specific or granular, they generally relied on their own analyses when making strategic, operational, or resource planning decisions. For example, FEMA officials agreed with the QHSR insights that climate change and natural disasters were a serious threat to homeland security; however, officials said they have been monitoring these issues through FEMA’s multi-year Strategic Foresight Initiative. Further, officials provided several reasons why they do not use or are unable to use the stand-alone risk products. For example, officials from one component questioned the quality of the threat information and said they would like to see a more specific ranking of threats, while officials from another component said the results were too broad. Finally, officials from 3 of the 6 components were unaware of the stand-alone risk documents.

DHS officials stated that the QHSR presented a list of key homeland security areas and explained that they chose not to rank or prioritize within those areas because of concerns on the part of senior leadership, among other issues. For instance, former DHS officials who led the development of the HSSEA said that there were some hazards that stood out as high risk across multiple consequence categories, but reported DHS did not rank the risks because they were unable to develop criteria for risk tiers (such as what would be high risk, medium risk, or low risk, for example). Further, officials were concerned that risk-ranked priorities might imply the need to make resource changes to particular areas, when risk is just one of many inputs for decision-making.

Former DHS officials who managed the QHSR risk assessment also said that conducting a DHS-wide risk characterization was a challenge on many levels. It required substantial dedicated resources and consensus on what the analysis would describe (e.g., which scenarios and models should be used). The officials added that there are not enough resources at DHS to put sufficient effort into an enterprise-wide risk assessment that includes prioritization of risk outcomes. Further, the officials acknowledged that risk communication—including communicating the process for and implications of prioritized risk-rankings—is a challenge. Officials said that integrating risk-informed decision-making throughout the department will require continued growth of DHS’s risk analysis community. The National Academies 2010 review of DHS’s approach to risk management echoed these statements in their recommendations to DHS, which included building a strong risk culture at DHS and incorporating robust social science capabilities into risk analyses and risk management practices.

DHS officials said that they used the HSSEA results and QHSR to broadly inform department-level strategic planning. For example, officials said the System Mapping Initiative helped inform the development of the Southern Border and Approaches Campaign Plan and Strategy. Officials added that the risk results helped inform the selection of strategic priorities in the DHS Strategic Priorities Framework and were incorporated into the annual budget guidance DHS provides to components. Because the Strategic Priorities Framework and annual budget guidance are considered internal deliberative work products, we were unable to review the complete documents in order to determine the extent to which the HSSEA and QHSR risk results were included. However, officials from ICE reported that the Strategic Priorities Framework was useful for understanding DHS’s position on where to accept risk.

The 2013 risk management supplement to the NIPP states that comparing and prioritizing the risks faced by different entities helps identify where risk mitigation is most needed and determines and helps justify the selection of the most cost-effective risk management options. This risk prioritization supports resource allocation decisions, guides investments in these programs, and highlights the measures that offer the greatest return on investment. According to the NIPP, the risk prioritization process involves aggregating, combining, and analyzing risk assessment results to determine which assets, systems, networks, sectors, or combinations of these face the highest risk so that risk management priorities can be established. It also provides the basis for understanding potential risk-mitigation benefits that are used to inform planning and resource decisions. The QHSR risk assessment describes a wide range of homeland security challenges and is a valuable step toward using risk information to prioritize and select risk management activities. While the QHSR makes use of risk information, it provides limited information to support resource allocation or risk-based strategic planning, such as a relative risk ranking of the department’s strategic priorities. Without determining and communicating prioritized risk analysis outcomes, DHS is missing an opportunity to more efficiently implement programs, strategies, and policies to mitigate risk and is limited in its ability to identify the resources required for addressing different levels and types of risks.

DHS Budget Request and Performance Measures Align with QHSR Mission Goals

DHS’s Budget Request Aligns with the QHSR, and DHS is Taking Actions to Account for its Spending by Mission Area

The Department’s annual budget request was generally presented in alignment with the QHSR mission areas, but DHS has faced challenges accounting for its spending by mission, which it is taking actions to address. Specifically, the manner in which DHS’s annual appropriation is structured does not directly track to the DHS missions, and DHS officials reported that significant budget disparities and inconsistencies between its components and their appropriations and programs have contributed to a lack of transparency, inhibited comparisons between programs, and complicated managerial decision making.

The President’s annual budget request for the department, as reflected in the DHS’s Budget in Brief for fiscal years 2015 and 2016, generally described funding priorities and requests at an organizational level (such as by component, office, and directorate), not by QHSR mission area. The Budget in Brief does, however, highlight select programs and activities that fall within each of the department’s QHSR mission areas. For example, the fiscal year 2016 Budget in Brief provided that within QHSR mission area 1—Preventing Terrorism and Enhancing Security— the department requested $3.7 billion for TSA screening operations, $101 million for radiological and nuclear detection equipment acquisition, and $94.5 million for Infrastructure Security Compliance funding to secure high risk chemical facilities, among other requests. The fiscal year 2016 Budget in Brief provided similar examples within each DHS mission area, as well as examples of requests for other departmental initiatives.

In general, DHS receives its funding through the enactment of an annual appropriations act. Historically, DHS’s annual appropriations have generally been organized into five titles, within which over 70 individual appropriation accounts and over 100 programs, projects, and activities (PPAs) are funded. The PPAs within DHS’s appropriation are generally not organized by mission-based programs, and according to DHS officials do not afford transparency or an easy understanding of how federal funds are being used. To provide increased visibility, comparability, and information on which to base resource allocation decisions, the report of the Committee on Appropriations of the House of Representatives accompanying the Department’s fiscal year 2015 appropriations bill directed DHS to develop a common appropriation structure (CAS) for implementation in the fiscal year 2017 budget cycle. According DHS’s fiscal year 2017 Budget in Brief, the President’s budget for the Department was submitted under a CAS that will improve DHS’s planning, programming, budgeting, and execution processes through a common framework consolidating its over 70 appropriations accounts into four standard appropriations and will better align programs to the DHS strategic missions and goals.

Officials from the Office of the Chief Financial Officer (OCFO) explained some of the potential benefits they hope to achieve through the CAS. For example, OCFO officials explained that while under the current system DHS is able to report broadly—such as on the amount spent on border security—it cannot report on specific obligations and expenditures within that category because the funding spans multiple agencies with different accounting systems. However, the CAS (in conjunction with another ongoing project called the Accounting Classification Structure), is intended in part to help DHS better bridge its budget requests with execution of the QHSR strategy. Specifically, officials said their top level of accounting will reflect the life cycle of funds, while the second level will be standardized to newly defined mission programs. Officials said this change will make it easier to understand and compare what is currently captured by 100-plus PPAs that do not have a direct link to the QHSR missions and goals. DHS currently tracks this information to some extent, but in what they referred to as a shadow system that is not sustainable over the long term.

Although DHS expects the CAS to improve accountability of funds and better align its PPA structure to focus on mission programs, DHS has reported that migrating over 70 appropriations accounts to four basic appropriation fund types will entail changes to the department’s PPA structure and that DHS has already faced challenges upgrading and integrating the many varied DHS financial systems, as well as maintaining oversight over the funding. DHS budget officials stated that DHS is working to manage these and other challenges through regular communication with component-level leadership and by briefing Congress regarding the budget and staffing levels needed to successfully complete the project.

DHS Budget Guidance Reflects QHSR Priorities; Providing Guidance to Components Sooner is an Ongoing Priority

DHS has taken action to improve its guidance to components for their annual budgeting processes—one of many actions underway as part of the department’s Unity of Effort initiative. Officials from each component we interviewed (CBP, USCG, FEMA, ICE, NPPD, TSA) said the department’s annual Resource Planning Guidance (RPG) mirrored the priorities listed in the QHSR and fiscal year 2014-2018 Strategic Plan. Further, officials from 5 of 6 components said that the RPG was an improvement over the previous budget guidance, known as the Integrated Planning Guidance, since the RPG’s development was more transparent and the priorities were clearer. However, officials from 5 of the 6 components said the most recent guidance—for the fiscal years 2017- 2021 budget cycle—was not issued with sufficient time to be incorporated into the budgeting process, which requires components to submit their budgets to DHS in the spring of each year. Specifically, officials said they did not receive the guidance until February 2015 for an April 2015 deadline. Officials said that this potentially limited their ability to staff issue teams to explore budget issues and to engage in more detailed planning and analysis. Component officials reported that receiving the guidance in the fall would be preferable. DHS policy officials are aware of this concern and issued its fiscal years 2018 through 2022 guidance in December 2015 and stated that the department plans to issue future guidance to components before the end of each calendar year.

DHS Implemented Performance Measures for Each QHSR Mission and Goal

DHS developed performance measures that link with each of the sixteen 2014 QHSR mission goals, as shown in table 2, thus meeting one of the attributes of successful performance measures—that they should link, or align, with an agency’s goals and missions.

We reported in 2011 that DHS had not developed performance measures for all of the 2010 QHSR missions, goals, and objectives. Specifically, DHS had developed performance measures for 13 of the 14 mission goals, having not yet developed performance measures for Goal 2.3, Disrupt and Dismantle Transnational Criminal Organizations. For the 2014 QHSR, as mentioned above, the number of mission goals increased to 16 (instead of 14), and DHS had established performance measures for all 16 mission goals—including Goal 2.3—as well as new measures related to new Mission 4 goals for cybersecurity. DHS’s Annual Performance Report, which presents DHS’s performance measures and applicable results aligned to missions, and provides the planned performance targets and priority goals, indicated that at the end of fiscal year 2014, 63 percent of all of DHS’s measures met their targets, and a review of trends in the results showed that 71 percent of the measures sustained or improved performance from fiscal year 2013. However, according to DHS, not all of the mission goals have credible outcome measures. We did not assess the quality or usefulness of the performance measures for the 2014 QHSR mission goals, as this was outside the scope of our review.

According to DHS SPAR and PA&E officials, both the 2010 and 2014 QHSR reports informally relied on feedback from DHS performance measures to inform changes to the QHSR. In addition, DHS has developed an approach relying on an existing process—referred to as the Strategic Review Process—that assesses whether changes are needed to broad DHS planning documents, such as the QHSR, based on performance measure results. In 2014, DHS conducted its first Strategic Review Process while the second QHSR was in its final clearance and publication phase. According to DHS officials, findings from the Strategic Review Process may help provide feedback for areas of focus and new measure development. Further, according to officials, the use of the DHS Annual Performance Report is just one of multiple resources DHS SPAR will use to inform development of the next QHSR.

DHS Expanded Outreach to Stakeholders but Could Improve Collaboration

DHS Sought QHSR Input from a Broader Range of Stakeholders

DHS took steps to expand its outreach in 2014 QHSR process to elicit input from a broader set of stakeholders, including additional nonfederal entities such as state, local, and tribal governments; private organizations and foreign governments. In addition to the seven federal agencies specified by the 9/11 Commission Act, DHS consulted directly with two additional federal agencies and doubled the number of stakeholder organizations that it contacted during the 2014 QHSR process. DHS also provided briefings to five foreign governments—Australia, Canada, Mexico, New Zealand, and the United Kingdom—and provided regular briefings to members of Congress, the Executive Office of the President, and other federal partners. In total, DHS estimates that it contacted about 40,000 homeland security stakeholders via email requesting their input and expertise in the 2014 QHSR process. Although DHS SPAR officials used multiple mechanisms to reach stakeholders during the 2014 QHSR process, including briefings, meetings and existing committee structures, it relied primarily on its electronic tools to elicit input from nonfederal stakeholders. Figure 8 provides an overview of the collaboration mechanisms used to engage stakeholders.

According to DHS officials, the electronic tools elicited the views of more than 2,000 registered participants and informed the analytical studies that ultimately shaped the final QHSR report. Given the volume and diversity of views among the stakeholders, DHS determined that using the electronic tool mechanism was the most efficient way to capture the majority of the input from the nonfederal entities for the 2014 QHSR review.

Time Constraints Present in First QHSR Persisted in Second QHSR

In 2011, we recommended that DHS provide more time for consulting with stakeholders during the QHSR process to help ensure that stakeholders have the time needed to review QHSR documents and provide input into the review. We recommended that DHS build this time into the department’s project planning for the 2014 QHSR. Yet, this recommendation was not implemented and the timeliness issue persisted in the 2014 QHSR, based on what QHSR stakeholders told us. In a survey we conducted of QHSR stakeholders, we asked whether DHS provided a clear overview of the timeframes and stakeholder responsibilities during each of the four phases of development. Stakeholders varied in their responses from about fourteen percent reporting that they did not receive a clear overview of the timeframes and responsibilities in phase 1 to 34 percent in phase 4 of the QHSR process. Stakeholder narrative responses also indicate that these timeframes shifted and that overall time constraints persisted throughout the QHSR process. For example, in the narrative portion of our survey, almost all of the DHS respondents (14 out of 16) who replied to a question on timeliness stated that time constraints during the consultation process either led to disruptions or may have hindered their ability to properly identify and consult with appropriate subject matter experts within their agency. One respondent noted that accelerated timeframes negatively impacted their ability to ensure substantive engagement with the state, local, and tribal leaders while another noted that “arbitrary” and “unpredictable” timeframes resulted in time away from their mission. In addition, in our interviews with federal stakeholders, officials from five of nine federal agencies stated that involving the agencies earlier in the process could help alleviate compressed timeframes in the final review phase.

According to program management standards, stakeholder and program time management are recognized practices, among others, for operating programs successfully. Because stakeholders are defined as those whose interests might be affected by the program outcomes and play a critical role in the success of any program, stakeholder consultations should include an active exchange of accurate, consistent and timely information that reaches all relevant stakeholders. Time management is necessary for program components and entities to keep the overall program on track and produce a final product. According to SPAR officials, the time constraints for the 2014 QHSR were related to a combination of factors such as a communication delay associated with the government shutdown and the change in DHS leadership in 2013, delays in the report vetting process, and inadequate planning in the early stages of the review. For example, the delay in the release of the terms of reference document led to delays in other areas such as the establishment of study groups and ultimately delayed the efforts to conduct outreach to stakeholders. DHS officials responsible for conducting stakeholder outreach expressed frustration with the delays and noted that they were unable to move forward with planned outreach activities. In addition, DHS SPAR officials acknowledged that engaging the stakeholder community early and regularly needs to be built into the planning for the review.

We continue to believe that taking steps to ensure stakeholders have the time necessary to provide quality input is critical for DHS to incorporate their expertise and perspectives. We recognize the possibility of leadership changes and report vetting delays, but it is precisely because of such contingencies that planning for and building in sufficient time would better ensure that all key stakeholders can fully participate in the event of such delays. As a result of time constraints, the QHSR process may not have benefitted from the full participation of all relevant stakeholders and may have missed opportunities to incorporate their perspectives.

Improved Use of Collaboration Mechanisms Could Benefit Stakeholder Consultations

DHS Did Not Provide Sufficient Feedback Opportunities During Meetings

According to QHSR planning documents and DHS Policy officials, obtaining and incorporating the perspectives of the entire homeland security enterprise, including relevant federal and nonfederal representatives was a key step in the development of the 2014 QHSR. Federal and nonfederal survey respondents indicated, however, that QHSR stakeholder outreach meetings did not allow for interactive exchange between stakeholders and DHS officials. In the narrative responses to our survey, 43 of 61 respondents to one question in our survey stated that collaboration with stakeholders could be improved. These respondents included federal and nonfederal stakeholders. For example, one survey respondent noted that stakeholders were asked to react to information provided by DHS rather than participating in formulating the approach and execution of the studies.

In addition to indications of collaboration challenges in our stakeholder survey results, during our interviews with federal departments and agencies, six of nine federal representatives stated that they felt DHS’s communication with them was “one-way” and led to an overall sense that stakeholder input was not valued or genuinely sought after.

DHS also held working meetings with internal DHS QHSR stakeholders, including senior and component leadership, to elicit input and feedback. However, the After Action Report also stated that some stakeholders believed that meetings included issues that had already been resolved or decided upon by DHS leadership. Specifically, the report noted that the DHS components viewed the meetings as opportunities to “check the box” that internal DHS stakeholders were consulted instead of true opportunities for input. The report recommended that future QHSR stakeholder consultation sessions follow a structure where leaders ask for information, groups present findings for discussion, and leaders review and provide feedback until a resolution is reached.

Project management standards state that effective stakeholder engagement involves gaining and maintaining stakeholder “buy in” for the program’s objectives, benefits, and outcomes throughout the effort. Specifically, program managers should employ communication methods that target specific stakeholder needs, expectations, and wants and also consider interactive “two way” communication since it is the most efficient way to ensure common understanding by all participants on specified topics.

DHS SPAR officials stated that although they initially planned to employ interactive meetings, particularly to reach the nonfederal stakeholders, staff, resource, and time constraints restricted DHS’s ability to engage in a fully collaborative effort and hindered its ability to employ other mechanisms that would have enabled more interactive feedback. DHS used mechanisms such as existing groups within DHS and events to connect with nonfederal stakeholders, but DHS officials stated that these events did not provide interactive feedback with the audience. For example, DHS briefed stakeholders from the private sector via meetings and although these meetings are helpful in contacting relevant stakeholders and informing them of QHSR issues, these sessions did not allow for stakeholder responses and exchange. DHS SPAR officials further acknowledged that the effort to reach out to stakeholders via electronic tools became more of an information sharing exercise instead of a means for interactive feedback. In addition, DHS’s 2014 QHSR After Action Report noted that the online tools were used to validate study findings instead of informing them as originally planned.

By not fostering interactive communication with federal and nonfederal stakeholders, DHS may have missed opportunities to fully engage the entire homeland security enterprise and thereby may not have fully informed the QHSR effort. A means for engaging both federal and nonfederal stakeholders in interactive communication could help ensure common understanding across the enterprise and elicit more robust contributions for stakeholders.

DHS Did Not Sufficiently Clarify Roles and Responsibilities for Detailees

In order to facilitate collaboration and information sharing among the internal DHS stakeholders, DHS Policy requested and received detailee staff at the supervisory level from each of the DHS internal components to serve a 6 month assignment with the QHSR core team led by DHS SPAR. Specifically, detailees were to serve as primary analysts supporting the QHSR studies and perform liaison activities between DHS SPAR and component leadership. According to DHS SPAR officials, the detailee program was a useful mechanism in ensuring that DHS components had a voice in the QHSR process. DHS SPAR officials added that the detailees provided valuable insights and contributions to the QHSR process. However, in the narrative portion of our survey and in our interviews with DHS components, respondents expressed concerns regarding the purpose and utilization of these staff. For example, DHS officials at 4 of 9 components told us that the staff detailed to support the QHSR effort either could have used additional clarity or did not have a clear understanding of their roles and responsibilities. One of the detailees stated that they had not been used to communicate information between the component and DHS SPAR. In addition, the DHS After Action Report acknowledged that some detailees were not utilized effectively, seemed to lack clear tasks to perform on a regular basis, and did not have subject matter expertise needed to support the studies. For example, the report noted that some detailees struggled comprehending the terminology used during the studies and therefore were confused as to how they could best support the effort. The After Action report stated that the components receive more advanced notice on the type of detailee that would be needed to support the QHSR process and consider identifying the study topics in advance of requesting the detailees so that the components can provide detailees with relevant expertise.

According to key principles of inter-agency collaboration, identifying and clarifying the roles and responsibilities of staff, such as component detailees, by ensuring that they have appropriate knowledge, skills, and abilities, can strengthen collaboration. Such clarification would better enable DHS to derive the benefit of obtaining homeland security enterprise-wide input and feedback. However, DHS SPAR officials stated that some of the confusion about detailee roles and responsibilities may have been because SPAR received a detailee at the supervisory level from each component, but later recognized that it needed individuals with relevant analytical skill sets to inform the QHSR—not just supervisory skills. Although the initial DHS Policy letter requesting detailees from components specified the need for analytical skills, it did not specify the types of or areas of expertise needed. The After Action Report also stated that by requesting study topics earlier there would be additional time to identify detailees with relevant subject matter expertise. SPAR officials stated that in the future, they would consider specifying the types of expertise needed from the component staff in advance of their request. Without recruiting detailees with the appropriate subject matter expertise and identifying and clarifying the roles and responsibilities, component stakeholders may not have the information that they need to fully participate and be invested in the process, and may thereby not be fully committed to the QHSR effort.

Conclusions

The homeland security threats to the nation are continuously evolving and in recent years have included cyber security breaches, mass shootings, and natural disasters that have devastated portions of the United States. In order to effectively respond to the nation’s changing security conditions, DHS must regularly review and refine the nation’s homeland security strategy, missions, and goals. Both the first and second Quadrennial Homeland Security Reviews were massive undertakings involving considerable time and resources on the part of DHS, as well as the input of numerous federal agencies, state and local government entities, and academics. In developing its second QHSR, DHS took important steps toward assessing homeland security risks and also sought to engage a wider audience of stakeholders in developing and reviewing the strategy. DHS officials said the QHSR risk assessment was highly complex and they faced limited time, personnel, and available data with which to conduct and document the assessment. However, not fully documenting the assessment and its methodology lessens the defensibility of the results and limits the assessment’s reproducibility for future QHSRs. Furthermore, these data and methodological limitations, along with concerns on the part of senior leadership with using risk assessment results to support a relative risk-ranking of the department’s strategic priorities, resulted in DHS missing an opportunity to use the QHSR to more efficiently implement programs, strategies, and policies to address different levels and types of risks. Last, insufficient forums for multi-directional collaboration and DHS SPAR’s unclear expectations of component detailees resulted in DHS missing opportunities to engage with the full range of homeland security stakeholders capable of informing the QHSR effort. For its next QHSR, DHS has an opportunity to build on its progress to date in order to ensure its missions, goals, and strategy addresses the most significant homeland security risks and is representative of the full breadth of homeland security stakeholders.

Recommendations for Executive Action

To ensure the quality of the risk assessments used to inform its future QHSR processes, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to ensure future QHSR risk assessment methodologies reflect key elements of successful risk assessment methodologies, such as being:

Documented, which includes documenting how risk information was integrated to arrive at the assessment results,

Reproducible, which includes producing comparable, repeatable

Defensible, which includes communicating any implications of uncertainty to users of the risk results.

To enable the use of risk information in supporting resource allocation decisions, guiding investments, and highlighting the measures that offer the greatest return on investment, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to refine its risk assessment methodology so that in future QHSRs it can compare and prioritize homeland security risks and risk mitigation strategies.

To ensure proper management of the QHSR stakeholder consultation process, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to identify and implement stakeholder meeting processes to ensure that communication is interactive when project planning for the next QHSR.

To ensure proper management of the internal QHSR stakeholder consultation process, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to clarify component detailee roles and responsibilities when project planning for the next QHSR.

Agency Comments and Our Evaluation

We requested comments on a draft of this report from DHS. On April 4, 2016, DHS provided written comments, which are reprinted in appendix V. DHS concurred with our four recommendations and described actions planned to address them. With regard to our first recommendation that DHS ensure future QHSR risk assessment methodologies reflect key elements of successful risk assessment methodologies—such as being documented, reproducible, and defensible—DHS stated that it believed that the 2013 Homeland Security National Risk Characterization (HSNRC) was complete, documented, reproducible, and defensible, consistent with the National Infrastructure Protection Plan guidance. DHS further stated that the HSNRC was developed using sound risk analysis methodologies informed by well-established analytical protocols and reproducible data provided by multiple DHS and component programs and subject matter experts, and that analysis was reviewed by the DHS Risk Executive Steering Committee and documented throughout the development of the HSNRC.

However, our review of DHS’s HSNRC documentation, as well as documentation on the Homeland Security Strategic Environment Assessment (of which the HSNRC is a part), found that DHS did not fully document how its risk results (from both its quantitative and qualitative analyses) were synthesized to generate its list of standout hazards and included only a limited description of data DHS relied on to generate its results. Further, DHS did not provide a description of how any data or methodological limitations should be interpreted by users of the results. Our review of DHS’s documentation also found that DHS did not provide an explanation of how all of the various Homeland Security Strategic Environment Assessment elements—the system maps, the risk ranking from the HSNRC, the trends analysis, or the threat reports—were combined to generate the QHSR risk narrative or other risk findings. DHS provided an explanation that its various analyses were combined in order to generate insights, but did not provide descriptions of the weighting factors, assumptions, and subjective judgments that went into synthesizing what DHS referred to as static and dynamic risk. In addition, beyond informing a strategic context, it is unclear how (if at all) decision- makers should interpret and use the QHSR risk results. Incorporating the key principles of a successful risk assessment methodology can help DHS demonstrate the basis of its risk management strategies and help ensure their validity for guiding QHSR mission and goals.

Regarding our second recommendation that DHS refine its risk assessment methodology so that in future QHSRs it can compare and prioritize homeland security risks and risk mitigation strategies, DHS stated that it will continue to refine its risk assessment methodology and expand its use of comparative risk analysis in the next HSNRC. Regarding our third recommendation that DHS identify and implement stakeholder meeting processes to ensure that communication is interactive when project planning for the next QHSR, DHS stated it is developing an extensive stakeholder outreach plan to include leveraging the Homeland Security Information Network and its access to a broad community of federal, state, and local stakeholders across the homeland security enterprise. The outreach plan is to include using on-line tools, in- person subject matter expert interviews, and extensive interagency coordination to facilitate information exchanges, document reviews, and feedback throughout the analysis and review phases of the next QHSR. In regards to the fourth recommendation that DHS clarify component detailee roles and responsibilities when project planning for the next QHSR, DHS stated that it will request DHS directorate and component detailees with specific skill sets and provide clearly defined timelines and objectives to support the QHSR. DHS also provided technical comments, which we incorporated as appropriate.

We also requested comments on a draft of this report from the Departments of Agriculture, Defense, Health and Human Services, Justice, State, Transportation, Treasury, and Veterans Affairs, the Environmental Protection Agency, and the Office of the Director of National Intelligence. The Department of State provided comments in an email received March 31, 2016, emphasizing the importance of risk- based resource allocation as well as the importance of linking operations and performance metrics to risk information. In an e-mail received from a departmental liaison on March 23, 2016, the Department of Agriculture indicated that it had no comments on the report. In e-mails received from departmental liaisons on March 29, 2016, the Department of Transportation and Department of Veterans Affairs indicated that they had no comments on the report. In e-mails received from departmental liaisons on March 30, 2016, the Department of Health and Human Services, Department of Defense, and the Office of the Director of National Intelligence indicated that they had no comments on the report. In an e-mail from a departmental liaison on April 1, 2016, the Department of Justice indicated that it had no comments on the report. In an e-mail received on April 4, 2016 from an agency liaison, the Environmental Protection Agency indicated that it had no comments on the report. In an e-mail received on April 8, 2016, from the Department of the Treasury’s Director for Emergency Programs the department indicated that it had no comments on the report.

We are sending copies of this report to the Secretaries of Agriculture, Defense, Health and Human Services, Homeland Security, State, Transportation, Treasury and Veterans Affairs; the Attorney General; the Director of National Intelligence; Administrator of the Environmental Protection Agency; and selected congressional committees. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (213) 830-1011 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: 9/11 Commission Act Quadrennial Homeland Security Review (QHSR) Reporting Elements

Of the nine reporting elements specified in the Implementing Recommendations of 9/11 Commission Act of 2007 (9/11 Commission Act) for inclusion in the QHSR report, the Department of Homeland Security (DHS) addressed five and did not fully address four through the 2014 QHSR report, finalized in June 2014, and the DHS Fiscal Years 2014-2018 Strategic Plan (Strategic Plan), released in December 2014, as shown in table 4. Elements DHS addressed in those documents included a description of homeland security threats, discussions of the status of cooperation among federal agencies and between the federal government and state, local, and tribal governments, an explanation of underlying assumptions for the QHSR report, and other matters considered important. Elements not fully addressed included a prioritized list of homeland security missions, a description of the budget plan required to execute the full range of missions, and an assessment of the alignment of DHS with the QHSR missions.

DHS officials agreed with our overall assessment, but asserted that the specifications for some of the elements we found as not fully addressed were reported through other relevant documents. For example, according to DHS officials, the specification for identifying a budget plan was met through the FY2015-2019 Future Years Homeland Security Program report, published on August 11, 2014. DHS also stated that alignment of human resources systems was addressed in DHS’s FY2015-2019 Human Capital Strategic Plan. For the purposes of evaluating whether DHS addressed the QHSR reporting elements, we limited our review to the QHSR report and DHS’s Strategic Plan, which references the QHSR report and further articulates homeland security missions, goals and strategies. We did not review any additional documents that DHS had published pertaining to issues identified for inclusion in the QHSR report, like the budget or human resources-related matters. Therefore, we did not evaluate whether DHS addressed the reporting requirements through means outside of these two documents.

The statute establishing the QHSR provides that each such review “shall be a comprehensive examination of the homeland security strategy of the Nation, including recommendations regarding the long-term strategy and priorities of the Nation for homeland security and guidance on the programs, assets, capabilities, budget, policies, and authorities of the Department.” The statute further requires that DHS submit to Congress a report regarding the QHSR, and identifies specific elements the report is to include. Our review of the QHSR report and the Strategic Plan concluded that while DHS had addressed five of the elements mandated for inclusion, it did not fully address four of the elements. Consequently, to the extent Congress anticipated the submission of a single comprehensive QHSR report that fully addressed each of the nine elements specified in the 9/11 Commission Act, rather than multiple DHS products that may collectively address each element, the report submitted to Congress on June 18, 2014, falls short. In addition, DHS issued the QHSR report after December 31, 2013, the date contemplated in the 9/11 Commission Act for DHS to report on the results of the quadrennial review conducted in 2013. According to DHS officials, DHS released the QHSR report after this date because leadership changes, specifically the appointment of a new Homeland Security Secretary in October 2013, occurred prior to the completion of the 2014 QHSR and thus extended the report’s review time frames.

Appendix II: Scope and Methodology

To determine the extent to which the Department of Homeland Security (DHS) has examined and used risk information to inform the Quadrennial Homeland Security Review and its implementation, we analyzed DHS documentation on its risk analysis process and results for the Homeland Security Strategic Environment Assessment (HSSEA)—the risk analysis conducted for the QHSR. We reviewed DHS’s Current and Future Strategic environment reports, the Systems Mapping Initiative report, the Homeland Security National Risk Characterization (HSNRC), and classified HSSEA results. We also reviewed the 2012 and 2013 Worldwide Threat Assessment of the U.S. Intelligence Community, prepared by the Office of the Director of National Intelligence (ODNI), which DHS reported relying on when assessing threat information for the HSSEA. In addition to the HSSEA reports themselves, we analyzed all available supporting documentation provided by DHS on the development of these reports, including briefings, videos, meeting summaries, and descriptions of HSSEA processes and results. We evaluated the sufficiency of this documentation and the overall completeness of the HSSEA risk assessment using the standards set forth in the 2013 National Infrastructure Protection Plan (NIPP) and its Supplemental Tool on Executing a Critical Infrastructure Risk Management Approach. The NIPP states that in order to be complete, a risk assessment must consider threat, vulnerabilities, and consequences; must be sufficiently documented; must be reproducible; and must be defensible. In order to be sufficiently documented, the methodology and the assessment must clearly document what information is used and how it is synthesized to generate a risk estimate. Any assumptions, weighting factors, and subjective judgments need to be transparent to the user of the methodology, its audience, and others who are expected to use the results. The types of decisions that the risk assessment is designed to support and the timeframe of the assessment (e.g., current conditions versus future operations) should be given. We also looked to a 2010 review by the National Academies of Science on DHS’s Risk Management Approach, as well as our previous work evaluating DHS’s approach to risk assessment, in order to ensure that we used a consistent approach to evaluating DHS’s assessment efforts. To supplement and clarify DHS’s written documentation on the HSSEA, we interviewed current and former DHS officials from the Office of Strategy, Plans, Analysis, and Risk (SPAR) who were responsible for developing the QHSR risk analyses. We asked officials to discuss the HSSEA process and how results were translated into the QHSR, including the extent to which uncertainty in the risk estimates was incorporated into the final results and communicated to users of the results—a NIPP aspect of a defensible risk assessment. Further, we evaluated DHS’s use of risk information in the QHSR to risk management guidance in the NIPP and DHS’s Risk Management Fundamentals, as well as our prior work on key characteristics for risk assessment and management. Last, we met with the authors of a December 2015 paper published in Homeland Security Affairs on a comparative ranking of homeland security hazards in order to discuss current challenges and developments in the field of homeland security risk analysis.

To determine the extent to which DHS has aligned its budget and performance measures with the mission goals in the QHSR, we analyzed DHS documents related to the 2014 QHSR and the fiscal years 2014- 2018 Strategic Plan, including: DHS Budget in Brief documents from 2015 through 2017 to determine DHS’s budget priorities since issuance of the first QHSR; fiscal years 2012 through 2015 Future Years Homeland Security Program (FYHSP) reports; excerpts from DHS’s fiscal years 2017-2021 and 2018-2022 resource planning guidance to determine the extent to which budget guidance reflects the DHS missions and goals; and DHS reports on its progress implementing a new common appropriations structure. Further, we reviewed Congressional Research Services reports on DHS’s appropriations from fiscal year 2015 and 2016. We also interviewed DHS officials from the Office of the Chief Financial Officer, Office of Strategy, Plans, Analysis, and Risk (SPAR), and from U.S. Customs and Border Protection (CBP), U.S. Coast Guard (USCG), Federal Emergency Management Agency (FEMA), U.S. Immigration and Customs Enforcement (ICE), National Protection and Programs Directorate (NPPD), and Transportation Security Administration (TSA). We selected these components based on their share of the overall DHS fiscal year 2015 budget, breadth of homeland security responsibilities, and other factors. Views from these components are not generalizable to all DHS components. We interviewed component officials to determine the extent to which they used DHS guidance in developing their annual budget requests and the extent to which such guidance reflects the QHSR missions and goals. With respect to performance measures we reviewed DHS’s Annual Performance Plans from fiscal years 2013-2015 and 2014-2016, reviewed our prior work on key attributes of successful performance measures, reviewed the GPRA Modernization Act of 2010 (GPRAMA), and interviewed DHS officials from SPAR and the office of Program Analysis and Evaluation (PA&E) and from CBP, USCG, FEMA, ICE, NPPD, and TSA components in order to determine the extent to which DHS developed performance measures that aligned with mission goals. We compared DHS’s process for monitoring and measuring performance to internal control standards and our prior work on best practices for implementation of strategies and initiatives.

To determine the extent to which DHS consulted with stakeholders in developing the QHSR, we distributed a web-based survey to 222 QHSR stakeholders identified by DHS based on 2014 QHSR participation. The stakeholders included representatives from 9 federal departments and agencies; 12 DHS components, directorates and offices; about 20 state, local, tribal, private and academic organizations. The stakeholders DHS identified did not include those who participated solely online and, as a result, we did not include these individuals within the scope of our review. We conducted 5 pretests with respondents from DHS, other federal agencies, and related think tanks to verify that (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place an undue burden on respondents, (4) the information could feasibly be obtained, and (5) the survey was comprehensive and unbiased. We made changes to the content and format of the questionnaire after the pretests based on the feedback we received. Before we administered the survey, we revised the questionnaire to reflect comments from an independent reviewer within GAO.

We identified 40 respondents that either had an undeliverable email address, were no longer employed with the federal government and/or did not participate in the 2014 QHSR. We removed these stakeholders from our final distribution list, resulting in about 180 possible respondents out of the original 222 that were identified by DHS. Questionnaires were completed by 93 individual stakeholders overall, resulting in a response rate of fifty-one percent.

Since the survey is not based on a sample, it does not have sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as non-sampling errors. For example, difficulties in interpreting a particular question, sources of information available to respondents, or entering data into a database or analyzing them can introduce unwanted variability into the survey results. We took steps in developing the questionnaire, collecting the data, and analyzing them to minimize such non-sampling error. For example, social science survey specialists designed the questionnaire in collaboration with GAO staff with subject matter expertise. Then, we pre tested the draft questionnaire to ensure that the questions were relevant, clearly stated, and easy to understand. When we analyzed the data, an independent analyst checked all computer programs. Since this was a web-based survey, respondents entered their answers directly into the electronic questionnaire, eliminating the need to key data into a database and thereby minimizing error.

We also asked open-ended questions regarding the QHSR stakeholder consultation process, including ways in which DHS elicited stakeholder input, how revisions to timeframes may have impacted stakeholder responsibilities, any challenges in collaborating with DHS and suggestions for improving future QHSRs. Of the 93 individual stakeholders that responded to our survey, 74 respondents provided narrative comments. We analyzed these comments to determine common benefits and challenges they identified regarding DHS consultations during the QHSR. The comments received from these respondents are not generalizable to the entire group of stakeholders, but the feedback provided insights into stakeholder perspectives on how QHSR stakeholder consultations were conducted and how they could be improved. We compared DHS’s stakeholder outreach efforts to project management standards.

To assess the extent to which the 2014 QHSR and fiscal years 2014- 2018 Strategic Plan reports addressed reporting elements listed in the 9/11 Commission Act, we determined the extent to which each element was addressed in the reports. To accomplish this determination, three GAO analysts independently compared the QHSR and strategic plan to each of the nine reporting elements to determine whether each element was addressed, not fully addressed, or not addressed. In cases when the analysts disagreed, they reviewed and discussed their independent assessments to reach concurrence. We considered an element addressed if all portions of it were included in either the QHSR or strategic plan, not fully addressed if one or more but not all portions of the element were included, and not addressed if neither the QHSR nor the strategic plan addressed any part of the element. In addition, we interviewed DHS officials involved in the quadrennial review to determine DHS’s position on how the 9/11 Commission Act reporting requirements were implemented.

We conducted this performance audit from January 2015 to April 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Hazard Scope for the DHS Homeland Security National Risk Characterization

Terrorism/Intentional Acts Aircraft as a Weapon

Chemical / Biological Food Contamination Terrorism Attack

Appendix IV: Strategies Linked to Quadrennial Homeland Security Review Missions and Goals

The following table outlines the list of strategies linked to Quadrennial Homeland Security Review missions and goals depicted in figure 5.

Mission 1: PREVENT TERRORISM AND ENHANCE SECURITY Goal 1.1: Prevent Terrorist Attacks

Analyze, fuse, and disseminate terrorism information;

Deter and disrupt operations; Strengthen transportation security; and

Counter violent extremism.

Goal 1.3: Reduce Risk to the Nation’s Critical Infrastructure, Key Leadership, and Events chemical, biological, radiological, and nuclear materials and weapons. Enhance security for the nation’s critical infrastructure from terrorism and criminal activity; and Protect key leaders, facilities, and national special security events.

Goal 2.3: Disrupt and Dismantle Transnational Criminal Organizations and Other Illicit Actors promote U.S. economic security and competitiveness. Identify, investigate, disrupt, and dismantle transnational criminal organizations; and

Disrupt illicit actors, activities, and pathways.

Promote lawful immigration; Effectively administer the immigration services system; and Promote the integration of lawful immigrants into American society.

Prevent unlawful entry, strengthen enforcement, and reduce drivers of unlawful immigration; and Arrest, detain, and remove priority individuals, including public safety, national security, and border security threats.

Enhance the exchange of information and intelligence on risks to critical infrastructure and develop real-time situational awareness capabilities that ensure machine and human interpretation and visualization; Partner with critical infrastructure owners and operators to ensure the delivery of essential services and functions; Identify and understand interdependencies and cascading impacts among critical infrastructure systems;

Collaborate with agencies and the private sector to identify and develop effective cybersecurity policies and best practices; and

Reduce vulnerabilities and promote resilient critical infrastructure design.

Goal 4.2: Secure the Federal Civilian Government Information Technology Enterprise

Coordinate government purchasing of cyber technology to enhance cost-effectiveness; Equip civilian government networks with innovative cybersecurity tools and protections; and Ensure government-wide policies and standards are consistently and effectively implemented and measured.

Goal 4.4: Strengthen the Cyber Ecosystem

Deter, disrupt, and investigate cybercrime.

Drive innovative and cost effective security products, services, and solutions throughout the cyber ecosystem;

Conduct and transition research and development, enabling trustworthy cyber infrastructure;

Develop skilled cybersecurity professionals; Enhance public awareness and promote cybersecurity best practices; and Advance international engagement to promote capacity building, international standards, and cooperation.

Mission 5: STRENGTHEN NATIONAL PREPAREDNESS AND RESILIENCE Goal 5.1: Enhance National Preparedness

Goal 5.2: Mitigate Hazards And Vulnerabilities

Goal 5.3: Ensure Effective Emergency Response regulation, resilient design, effective mitigation, and disaster risk reduction measures; and Prevent incidents by establishing, and ensuring compliance with, standards and regulations. Provide timely and accurate information;

Conduct effective, unified incident response operations; Provide timely and appropriate disaster assistance; and Ensure effective emergency communications. Ensure continuity and restoration of essential services and functions; and Support and enable communities to rebuild stronger, smarter, and safer.

Appendix V: Comments from the Department of Homeland Security

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ben Atwater (Assistant Director), Nanette Barton, Chuck Bausell, Jr., Charlotte Gamble, Susan Hsu, Thomas Lombardi, Amanda Miller, and Katrina Taylor made key contributions to this report.