Quadrennial Homeland Security Review: Enhanced Stakeholder Consultation and Use of Risk Information Could Strengthen Future Reviews
Published: 2011-09-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The United States continues to face a range of evolving threats, such as the 2010 attempted attack on the nation's air cargo system, that underscore why homeland security planning efforts are crucial to the security of the nation. The Implementing Recommendations of the 9/11 Commission Act of 2007 required the Department of Homeland Security (DHS) to provide a comprehensive examination of the U.S. homeland security strategy every 4 years. In response, DHS issued its first Quadrennial Homeland Security Review (QHSR) report in February 2010 and a Bottom-Up Review (BUR) report in July 2010, to identify initiatives to implement the QHSR. As requested, this report addresses the extent to which DHS (1) consulted with stakeholders in developing the QHSR, (2) conducted a national risk assessment, and (3) developed priorities, plans, monitoring mechanisms, and performance measures for implementing the QHSR and BUR initiatives. GAO analyzed relevant statutes and DHS documents on the QHSR and BUR processes and, in response to a request for comments on the processes, received comments from 63 of the 85 federal and nonfederal stakeholders it contacted. Their responses are not generalizable, but provided perspectives on the processes.
section_title: What GAO Found
paragraphs: DHS solicited input from various stakeholder groups in conducting the first QHSR, but DHS officials, stakeholders GAO contacted, and other reviewers of the QHSR noted concerns with time frames provided for stakeholder consultations and outreach to nonfederal stakeholders. DHS consulted with stakeholders--federal agencies; department and component officials; state, local, and tribal governments; the private sector; academics; and policy experts-- through various mechanisms, such as the solicitation of papers to help frame the QHSR and a web-based discussion forum. DHS and these stakeholders identified benefits from these consultations, such as DHS receiving varied perspectives. However, stakeholders also identified challenges in the consultation process. Sixteen of 63 stakeholders who provided comments to GAO noted concerns about the time frames for providing input into the QHSR or BUR. Nine DHS stakeholders, for example, responded that the limited time available for development of the QHSR did not allow DHS to have as deep an engagement with stakeholders. Further, 9 other stakeholders commented that DHS consultations with nonfederal stakeholders, such as state, local, and private sector entities, could be enhanced by including more of these stakeholders in QHSR consultations. In addition, reports on the QHSR by the National Academy of Public Administration, which administered DHS's web-based discussion forum, and a DHS advisory committee comprised of nonfederal representatives noted that DHS could provide more time and strengthen nonfederal outreach during stakeholder consultations. By providing more time for obtaining feedback and examining mechanisms to obtain nonfederal stakeholders' input, DHS could strengthen its management of stakeholder consultations and be better positioned to review and incorporate, as appropriate, stakeholders' input during future reviews. DHS identified threats confronting homeland security in the 2010 QHSR report, such as high-consequence weapons of mass destruction and illicit trafficking, but did not conduct a national risk assessment for the QHSR. DHS officials stated that at the time DHS conducted the QHSR, DHS did not have a well-developed methodology or the analytical resources to complete a national risk assessment that would include likelihood and consequence assessments--key elements of a national risk assessment. To develop an approach to national risk assessments, DHS created a study group as part of the QHSR process that developed a national risk assessment methodology. DHS officials plan to implement a national risk assessment in advance of the next QHSR, which DHS anticipates conducting in fiscal year 2013. DHS developed priorities, plans, monitoring mechanisms, and performance measures, but did not consider risk information in making its prioritization efforts. DHS considered various factors in identifying high-priority BUR initiatives for implementation in fiscal year 2012 but did not include risk information as one of these factors, as called for in GAO's prior work and DHS's risk management guidance, because of differences among the initiatives that made it difficult to compare risks across them, among other things. Consideration of risk information during future implementation efforts could help strengthen DHS's prioritization of mechanisms for implementing the QHSR, including assisting in determinations of which initiatives should be implemented in the short or longer term.
section_title: What GAO Recommends
paragraphs: GAO recommends that for future reviews, DHS provide the time needed for stakeholder consultations, explore options for consulting with nonfederal stakeholders, and examine how risk information could be considered in prioritizing QHSR initiatives. DHS concurred with our recommendations.

section_title: Letter
paragraphs: The United States continues to face a myriad of broad and evolving  threats, such as the October 2010 attempted attack on the nation’s air  cargo system, that underscore why the federal government places a high  priority on homeland security and efforts to coordinate security roles,  responsibilities, and activities across a wide variety of stakeholders,  including state, local, and tribal government; private sector;  nongovernmental; and international partners. The Implementing  Recommendations of the 9/11 Commission Act of 2007 (9/11  Commission Act) requires that beginning in fiscal year 2009 and every 4  years thereafter the Department of Homeland Security (DHS) conduct a  review that provides a comprehensive examination of the homeland  security strategy of the United States. According to the act, the review is  to delineate the national homeland security strategy, outline and prioritize  critical homeland security missions, and assess the organizational  alignment of DHS with the homeland security strategy and missions,  among other things. The act requires that DHS conduct the quadrennial  review in consultation with stakeholders, such as heads of federal  agencies; state, local, and tribal governments; private sector  representatives; and academics and other policy experts.
In February 2010, DHS issued its first Quadrennial Homeland Security  Review (QHSR) report, outlining a strategic framework for homeland  security to guide the activities of homeland security partners, including  federal, state, local, and tribal government agencies; the private sector;  and nongovernmental organizations. The report identified five homeland  security missions—(1) Preventing Terrorism and Enhancing Security,   (2) Securing and Managing Our Borders, (3) Enforcing and Administering  Our Immigration Laws, (4) Safeguarding and Securing Cyberspace, and  (5) Ensuring Resilience to Disasters—and goals and objectives to be  achieved within each mission. The QHSR report also identified threats  and challenges confronting U.S. homeland security, strategic objectives  for strengthening the homeland security enterprise, and federal agencies’  roles and responsibilities for homeland security. In addition to the QHSR  report, in July 2010 DHS issued a report on the results of its Bottom-Up  Review (BUR), a departmentwide assessment to implement the QHSR  strategy by aligning DHS’s programmatic activities, such as investigating  drug smuggling and inspecting cargo at ports of entry, and its  organizational structure with the missions and goals identified in the  QHSR. The BUR report described DHS’s current activities contributing to  (1) QHSR mission performance, (2) departmental management, and   (3) accountability. The BUR report also identified priority initiatives, such  as strengthening aviation security and enhancing the department’s risk  management capability, to strengthen DHS’s mission performance,  improve departmental management, and increase accountability. In  December 2010, we issued a report on the extent to which the QHSR  addressed the 9/11 Commission Act’s required reporting elements. We  reported that of the nine 9/11 Commission Act reporting elements for the  QHSR, DHS addressed three and partially addressed six. Elements  DHS addressed included a description of homeland security threats and  an explanation of underlying assumptions for the QHSR report. Elements  addressed in part included a prioritized list of homeland security missions,  an assessment of the alignment of DHS with the QHSR missions, and  discussions of cooperation between the federal government and state,  local, and tribal governments.
You asked us to review DHS’s QHSR, including DHS’s process for  conducting the review and for implementing the QHSR strategy. This  report addresses the following question: To what extent did DHS   (1) consult with stakeholders in developing the QHSR strategy; (2)  conduct a national risk assessment to develop the QHSR; and (3)  develop priorities, plans, monitoring mechanisms, and performance  measures for implementing the QHSR and BUR initiatives? This report  also provides information on the extent to which DHS’s strategic  documents and the 2010 National Security Strategy align with the QHSR  (see app. I).
To address these objectives, we analyzed DHS documents related to the  QHSR, BUR, and budget development processes, including the QHSR  report, BUR report, fiscal year 2012 budget request, and Fiscal Years  2012-2016 Future Years Homeland Security Program (FYHSP). We  identified criteria for evaluating these processes by analyzing our prior  reports on key characteristics of effective national strategies, key  practices for effective interagency collaboration, strategic planning,  performance measurement, and standards for internal control, among  others. Based on these reports, we identified those key practices and  characteristics applicable to quadrennial reviews, like the QHSR. The key  practices we identified were involving stakeholders in defining QHSR  missions and outcomes; defining homeland security problems and  assessing risks; including homeland security strategy goals, subordinate  objectives, activities, and performance measures; including resources,  investments, and risk management; including organizational roles,  responsibilities, and coordination across the homeland security  enterprise; and establishing DHS processes for managing implementation  of BUR initiatives. We vetted the key practices with our subject matter  experts—staff with legal and methodological expertise and experience  analyzing the Quadrennial Defense Review—and provided them to DHS  officials for review, and incorporated their comments as appropriate. As  we developed our report, we grouped these key practices into three  areas—stakeholder involvement, risk assessment, and implementation  processes for the QHSR and BUR initiatives.
To determine the extent to which DHS consulted with stakeholders in  developing the QHSR, we requested comments on the QHSR process  from 79 QHSR stakeholder organizations identified by DHS. The  stakeholders solicited by us for comments included 22 federal  departments and agencies; 10 state, local, and tribal organizations; 28  DHS components, directorates and offices; and 19 Quadrennial Review  Advisory Committee (QRAC) members. We also solicited comments  from 6 subject matter experts hired by DHS to facilitate QHSR study  groups. We received comments from 63 of the 85 stakeholders and  study group facilitators we contacted (74 percent), including 21 of 22  federal departments; 6 of 10 state, local, and tribal organizations; 26 of  the 28 DHS components, directorates, and offices; 7 of the 19 QRAC  members; and 3 of the 6 study group facilitators. We asked open-ended  questions regarding the QHSR stakeholder consultation process, such as  suggestions for improving future QHSRs, examples of positive ways DHS  involved stakeholders, and involvement in determining agency roles and  responsibilities listed in the QHSR report. We relied on respondents to  raise and comment on their views of the QHSR process; therefore we  could not determine whether respondents shared similar views or  identified similar benefits or challenges to the QHSR process unless  respondents identified them in their responses to our requests for  comments. We analyzed the comments provided by the 63 respondents  to determine common benefits and challenges they identified regarding  DHS consultations during the QHSR. We also conducted follow-up  interviews with 14 QHSR stakeholders that we selected based on their  responses, to obtain clarification of their responses to our requests for  comments. The comments received from these respondents are not  generalizable to the entire group of stakeholders, but the feedback  provided insights into stakeholder perspectives on how QHSR  stakeholder consultations were conducted and how they could be  improved. Further, we reviewed reports on the QHSR by the National  Academy of Public Administration (NAPA) and the QRAC, both of which  were based upon each organization’s collaboration experiences with DHS  in developing the QHSR report. We compared DHS’s stakeholder  consultation efforts to our prior work on effective practices for  collaboration and consultation.
To determine the extent to which DHS conducted a national risk  assessment to develop the QHSR, we analyzed risk analysis–related  documents produced as part of the QHSR process, such as DHS risk  assessment tools, and interviewed DHS officials responsible for  developing risk analyses for use at DHS. We compared DHS’s risk  assessment process in the QHSR to our prior work on key characteristics  for risk assessment as well as DHS risk analysis guidance documents.
To determine the extent to which DHS developed priorities,  implementation plans, monitoring mechanisms, and performance  measures, we analyzed DHS’s BUR implementation priorities and plans,  such as DHS’s fiscal year 2012 budget request; DHS monitoring  mechanisms, such as BUR initiative scorecards; our Standards for  Internal Control in the Federal Government; and DHS’s strategic and  management performance measures. We also interviewed DHS officials  responsible for managing and monitoring implementation of the BUR  initiatives. We compared DHS’s processes for prioritizing, monitoring, and  measuring implementation efforts to our prior work on key practices for  risk management and implementation and monitoring of strategic  initiatives. We also compared DHS’s performance measures for fiscal  year 2011 to our criteria on key attributes of successful performance  measures. A more detailed discussion of our scope and methodology is  contained in appendix II.
We conducted this performance audit from January 2011 through  September 2011 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DHS approached the 9/11 Commission Act requirement for a quadrennial  homeland security review in three phases. In the first phase, DHS defined  the nation’s homeland security interests, identified the critical homeland  security missions, and developed a strategic approach to those missions  by laying out the principal goals, objectives, and strategic outcomes for  the mission areas. DHS reported on the results of this effort in the  February 2010 QHSR report in which the department identified 5  homeland security missions, 14 associated goals, and 43 objectives, as  shown in figure 1. The QHSR report also identified a strategy for maturing  and strengthening the homeland security enterprise, with 18 associated  objectives. In the second phase—the BUR—DHS identified its component  agencies’ activities, aligned those activities with the QHSR missions and  goals, and made recommendations for improving the department’s  organizational alignment and business processes. DHS reported on the  results of this second phase in the July 2010 BUR report. In the third  phase DHS developed its budget plan necessary to execute the QHSR  missions. DHS presented this budget plan in the President’s fiscal year  2012 budget request, issued February 14, 2011, and the accompanying  Fiscal Year 2012-2016 FYHSP, issued in May 2011. DHS officials stated  that together, these three phases and their resulting reports and  documents address the 9/11 Commission Act requirement for the  quadrennial homeland security review.
DHS initiated the QHSR in August 2007. Led by the DHS Office of Policy, in July 2009 the department issued its QHSR terms of reference, outlining the framework for conducting the quadrennial review and identifying threats and assumptions to be used in conducting the review. Through  the terms of reference, DHS identified the initial four homeland security  missions to be studied, which were refined during the QHSR process— Counterterrorism and Domestic Security Management; Securing Our  Borders; Smart and Tough Enforcement of Immigration Laws; and  Preparing For, Responding To, and Recovering from Disasters—as well  as three other separate, nonmission study areas to be part of the  review—DHS Strategic Management, Homeland Security National Risk  Assessments, and Homeland Security Planning and Capabilities. The fifth  QHSR mission on Safeguarding and Securing Cyberspace was added  after DHS issued the terms of reference. A sixth category of DHS  activities—Providing Essential Support to National and Economic  Security—was added in the fiscal year 2012 budget request but was not  included in the 2010 QHSR report.
DHS established seven study groups for the QHSR, which were  composed of officials from across DHS offices and components. The  study groups were each led by a DHS official and facilitated by an  independent subject matter expert from the Homeland Security Studies  and Analysis Institute. These study groups conducted their analysis  over a 5-month period and shared their work products, such as outlines of  missions and assumptions, with other stakeholder groups in order to  develop goals and objectives for each mission. At the end of the study  group period, DHS senior leadership, including the Deputy Secretary of  Homeland Security, the General Counsel, and office and component  heads, met multiple times to review and discuss the study group  recommendations. The DHS Office of Policy consolidated the study  groups’ recommendations into a draft QHSR report and obtained and  incorporated feedback on the draft report from other federal agencies and  stakeholder groups, including the stakeholders listed in the 9/11  Commission Act, with which DHS was to consult in conducting the  QHSR. Agreement on the QHSR report’s final content was reached  between the Secretary for Homeland Security and senior White House  officials. DHS issued the final QHSR report in February 2010.
DHS initiated the BUR in November 2009. Each DHS directorate,  component, and office created an inventory of its activities and  categorized them according to the QHSR missions. For example, U.S.  Immigration and Customs Enforcement (ICE) identified one of its activities  as investigating human smuggling and trafficking, which it categorized  under the Securing and Managing our Borders QHSR mission. The BUR  resulted in a catalog of about 1,300 DHS activities organized under each  of the five QHSR missions or categorized as mission or business support  activities. DHS identified over 300 potential initiatives for increasing  mission performance and accountability and improving department  management, derived 43 priority initiatives from this list, and highlighted  them in the July 2010 BUR report. For example, under the Enforcing  and Administering our Immigration Laws mission, DHS identified as  priority initiatives improving DHS’s immigration services processes and  dismantling human-smuggling organizations (see fig. 2).
In addition, DHS categorized its 43 BUR initiatives according to whether  they require organizational, programmatic, policy, or legislative activities  in order to be implemented. DHS defines these categories as (1)  organizational, where implementation requires some type of departmental  reorganization (e.g., create a cybersecurity and infrastructure resilience  operational component within DHS); (2) programmatic, where  implementation requires budgetary activity, such as a funding increase  (e.g., increase efforts to detect and counter nuclear and biological  weapons and dangerous materials); (3) policy, where implementation  requires a policy decision but no additional funding (e.g., enhance the  department’s risk management capability); and (4) legislative, where  implementation requires a change in legislation or congressional approval  because DHS does not have the legislative authority to implement the  initiative (e.g., restore the Secretary’s reorganization authority for DHS  headquarters). According to DHS officials, some BUR initiatives require  one or more of these types of changes to be implemented, such as the  initiative to strengthen internal DHS counterintelligence capabilities, which  requires policy and programmatic changes.
DHS’s fiscal year 2012 budget request highlighted funding requests to  support projects and programs within each QHSR mission. For example,  for QHSR mission 1, Preventing Terrorism and Enhancing Security,  DHS’s fiscal year 2012 budget request includes requests for 18 projects  and programs to support that mission. These requests include items such  as $273 million for explosives detection systems at airports and   $12.4 million for enhanced watchlist vetting of airline passengers.  According to DHS officials, DHS intends to include longer-term project  and program funding plans for QHSR missions through annual iterations  of its FYHSP. For example, the Fiscal Year 2012-2016 FYHSP contains  initiatives and planned performance information aligned with the missions  of the department.
According to the 9/11 Commission Act, DHS is to report on the results of  its QHSR every 4 years with the next report due by December 31, 2013.  DHS plans to issue its next QHSR report in accordance with the act.
section_title: DHS Could Strengthen Stakeholder Consultations by Expanding Time Frames, Nonfederal Participation, and Role Definitions
subsections: section_title: DHS Used Various Mechanisms to Solicit Stakeholder Views during the QHSR Process
paragraphs: DHS solicited input from various stakeholder groups in conducting the  first QHSR. The 9/11 Commission Act required DHS to consult with seven  federal agencies in conducting the QHSR—the Departments of  Agriculture, the Treasury, Justice, State, Defense, and Health and Human  Services and the Office of the Director of National Intelligence. DHS  consulted with these agencies and also sought input from a range of  other stakeholders, including its directorates, offices, and components;  other federal agencies; and nonfederal governmental and  nongovernmental entities and representatives, such as state and local  governmental associations and individuals working in academia.
In obtaining input from these stakeholders, DHS used a variety of  mechanisms, such as multiagency working groups, solicitation of  homeland security research papers, and a web-based forum, referred to  as the National Dialogue, as shown in table 1.
We obtained comments from 63 stakeholders who DHS consulted with  through these mechanisms. The 63 stakeholders who responded to our  request for comments on the QHSR process noted that DHS conducted  outreach to them to solicit their views and provided opportunities for them  to give input on the QHSR. For example, DHS stakeholders, including its  directorates, offices, and components, reported participating in the QHSR  process by, for example, helping develop strategic outcomes and  measurable end states for the QHSR missions, assigning representatives  to the various QHSR study groups, and helping to draft QHSR report  language. Stakeholders from 21 federal agencies other than DHS and  its components that responded to our request for comments noted that  they provided input during the QHSR process by, among other things,  having representatives attend QHSR meetings, participating in sub- interagency policy committee meetings, and commenting on draft  versions of the QHSR report. Additionally, 6 nonfederal stakeholders  reported to us that DHS consulted with them by, for example, sending a  representative to association meetings, participating in conference calls to  discuss the QHSR, and holding stakeholder briefings to discuss QHSR  strategic goals, outcomes, and responsibilities.
DHS, QHSR stakeholders, and other entities, specifically the QRAC and  NAPA, that reviewed aspects of the QHSR identified various benefits  from DHS’s consultation efforts throughout the QHSR. For example, the  Deputy Assistant Secretary for Policy (Strategic Plans) stated that  stakeholder position paper submissions obtained at the beginning of the  QHSR process were beneficial in that study groups had stakeholder input  at the outset of the work. The Deputy Assistant Secretary also stated that  the National Dialogue was beneficial in that it gave DHS the ability to  gauge reactions to proposals for including information in the QHSR in real  time, as the National Dialogue represented a virtual discussion among  stakeholders. Further, 33 respondents to our request for comments on  the QHSR reported that one positive aspect of DHS’s consultations  during the QHSR was the range of stakeholders DHS contacted. Two  DHS stakeholders reported, for example, that DHS made extensive  efforts to involve a wide range of stakeholders and that involvement of  federal non-DHS agencies was beneficial in helping DHS obtain views on  the QHSR outside of the department. One DHS stakeholder noted that  the benefit of involving state, local, and private industry in the QHSR  study group discussions via the National Dialogue was that the study  groups were able to systematically consider viewpoints of the public  during the course of developing the QHSR mission goals and objectives.  The public perspectives offered different views than those provided by  DHS and other federal stakeholders. Similarly, 2 federal stakeholders  responded that the interagency meetings and the National Dialogue were  positive ways in which DHS involved stakeholders during the QHSR, and  that DHS’s consultations provided a mechanism for interagency  collaboration to discuss QHSR goal and objective areas. Additionally, one  QRAC member noted that DHS involved and coordinated well with  federal agencies; reached out reasonably well to state, local and tribal  organizations; included a large number of academics and other policy  experts; and gave the American public an opportunity to comment  through the National Dialogue.
Moreover, in its report on the QHSR, the QRAC noted that while not privy  to the details of all inputs received, the QHSR report represented a  synthesis of stakeholder consultations that was designed to set forth a  shared vision of homeland security in order to achieve unity of purpose  across the homeland security enterprise. In addition, with regard to the  National Dialogue, NAPA reported that by engaging stakeholders at all  levels, DHS was able to incorporate ground-level expertise and  specialized knowledge into the review. According to NAPA, by conducting  a process accessible to all interested parties, the National Dialogue  provided the opportunity to strengthen trust among stakeholders and  create potential buy-in for later implementation of policies and priorities  they helped to shape.
section_title: Time Constraints, Nonfederal Stakeholder Participation, and Definition of Stakeholders’ Roles Hindered QHSR Consultations
paragraphs: DHS consulted with a range of stakeholders through various  mechanisms, but DHS officials and stakeholders identified challenges  that hindered DHS’s consultation efforts in conducting the QHSR. These  challenges were (1) consultation time frames, (2) inclusion of nonfederal  stakeholders, and (3) definition of stakeholders’ roles and responsibilities.
According to DHS officials, the department consulted with stakeholders  primarily over a 5-month period—from May through September 2009— during the QHSR process. In response to our request for comments on  the QHSR process, 16 stakeholders noted concerns regarding the time  frames they had for providing input into the QHSR or BUR. Nine DHS  stakeholders, for example, responded that in their view, the limited time  available for development of the QHSR did not allow DHS to have as  broad and deep an engagement with stakeholders as DHS could have  experienced if more time had been allotted to stakeholder consultations.  DHS stakeholders also reported to us that DHS’s time frames for  conducting the BUR were short and that the BUR process was hampered  by an overly aggressive timeline for deliberation and decision making.  Two of the study group facilitators who responded to our request for  comments reported that in their view, stakeholders needed more time to  review draft work products and hold more discussions. Three federal  stakeholders suggested that the process be initiated earlier than it was for  the first QHSR to provide more time for DHS to consider and resolve  stakeholder comments, draft the report, and provide stakeholders with an  opportunity to review the draft report. One of these federal stakeholders  stated that more detail and other viewpoints would have been added to  the QHSR if DHS had conducted outreach earlier in the QHSR process  while another noted that it was difficult to keep up with the changes in the  QHSR draft report and therefore to fully participate in providing  comments. There were multiple drafts and no dialogue on how the  comments were incorporated, according to this stakeholder. This federal  stakeholder stated that more lead time in the provision of QHSR materials  would have allowed for stakeholders to better consider the information  and provide DHS with feedback. Two state and local associations  responded that more lead time for the arrangement of meetings and a  review of the complete QHSR report prior to its release would have been  helpful.
In addition, NAPA identified challenges associated with time frames for  conducting aspects of the QHSR. Specifically, in its report on the National  Dialogue, NAPA stated that the abbreviated turnaround time between  phases of the National Dialogue—approximately 3 weeks on average— resulted in very constrained time periods for the study groups to fully  review stakeholder feedback, incorporate it into the internal review  process, and use it to develop content for subsequent phases. NAPA  reported that for DHS to improve online stakeholder engagement it should  build sufficient time for internal review and deliberations into its timetable  for public engagement on the QHSR, and provide the public an  opportunity to see that it is being heard in each QHSR phase. Thus,  related to the National Dialogue, NAPA recommended that DHS build a  timetable that allows ample time for internal deliberations that feed  directly into external transparency. According to the Deputy Assistant  Secretary for Policy (Strategic Plans) at DHS, addressing NAPA’s  recommendations, in general, is part of the QHSR project planning to  begin during summer 2011 for the next QHSR. The official stated that  DHS is considering NAPA’s recommendations and is looking for  opportunities for additional stakeholder involvement during the next  QHSR.
DHS identified those stakeholders to be consulted and various  consultation mechanisms to be used prior to initiation of stakeholder  consultations, but planned the consultation time periods based on the  limited time available between when the QHSR process began and when  the report was due, contributing to the time frame concerns raised by the  16 QHSR stakeholders and NAPA. Our prior work on strategic studies  has shown that when federal agencies are defining missions and  outcomes, such as DHS did in developing the QHSR report, involving  stakeholders is a key practice. According to program management  standards, stakeholder and program time management are recognized  practices, among others, for operating programs successfully.  Stakeholder management defines stakeholders as those whose interests  may be affected by the program outcomes and that play a critical role in  the success of any program; it should ensure an active exchange of  accurate, consistent, and timely information that reaches all relevant  stakeholders. Time management is necessary for program components  and entities to keep the overall program on track, within defined  constraints, and produce a final product.
According to the Deputy Assistant Secretary for Policy (Strategic Plans)  at DHS, constrained time periods for stakeholder consultations are part of  the challenge of executing a time-limited process with a broad  stakeholder base, such as the QHSR. According to the Deputy Assistant  Secretary, longer time periods for stakeholder consultations could be  beneficial, but a tradeoff to consider is that the review as a whole would  be more time consuming. DHS officials determined time periods for  consultation by planning from the QHSR issuance date and then building  in stakeholder consultation periods for white paper solicitation and  receipt, the National Dialogue, and executive committee meetings.  Stakeholder consultation time frames were built into the QHSR project  plan, with planned time periods such as 23 days between white paper  solicitation notifications and the deadline for submissions from  stakeholders, which was dictated by the necessities of the December 31,  2009 issuance deadline. The National Security Staff set timelines for  report review by other federal agencies, according to the Deputy  Assistant Secretary. Moreover, this official said that setting target time  frames for stakeholder consultations during the next QHSR is something  that DHS plans to address during project planning. By considering ways  to build more time for stakeholder consultations into the timeline or target  time frames for the next QHSR, DHS could be better positioned to  manage stakeholder consultations and feedback received throughout the  process, including determining and communicating how much time  stakeholders will be given for providing feedback and commenting on  draft products. In addition, DHS could be better positioned to ensure that  stakeholders have the time needed for reviewing QHSR documents and  providing input.
DHS consulted with a range of stakeholders, including federal and  nonfederal entities, during the QHSR, and these consultations provided  DHS with a variety of perspectives for consideration as part of the QHSR  process. However, the department faced challenges in obtaining  feedback from nonfederal stakeholders. Our prior work on key practices  for performance management has shown that stakeholder involvement is  important to help agencies ensure that their efforts and resources target  the highest priorities. Involving stakeholders in strategic planning efforts  can also help create a basic understanding among the stakeholders as to  the agency’s programs and results they are intended to achieve. Without  this understanding successful implementation can be difficult because  nonfederal stakeholders help clarify DHS’s missions, reach agreement on  DHS’s goals, and balance the needs of other nonfederal stakeholders  who at times may have differing or even competing goals. As we have  previously reported, nonfederal entities have significant roles in homeland  security efforts. For example, state, local and private sector entities own  large portions of critical infrastructure in the United States and have  responsibilities for responding to and recovering from homeland security  incidents. Thus we have previously reported that it is vital that the public  and private sectors work together to protect these assets. Further, we  have reported on the need for federal and nonfederal entities to more  effectively communicate their emergency preparedness and response  roles, responsibilities, and activities. For example, we have reported  that effective public warning depends on the expertise, efforts, and  cooperation of diverse stakeholders, such as state and local emergency  managers and the telecommunications industry.
In responding to our request for comments, 9 stakeholders commented  that DHS consultations with nonfederal stakeholders, such as state, local,  and private sector representatives, could be enhanced. For example, 1  stakeholder noted that state, local and private sector representatives,  such as those with responsibility for securing critical infrastructure and  key resources, the maritime sector, and overseas interests, should be  further consulted during the next QHSR process. One federal stakeholder  noted that state and local involvement is critical for homeland security and  that a review of state and local readiness would be beneficial to  determine the gaps that would need to be filled at the federal level. DHS  could map out what state and local officials need in case of an emergency  and include the various federal agencies in these discussions. Further,  another stakeholder noted that DHS faced challenges in consulting  specifically with the private sector during the 2010 QHSR. DHS consulted  with private sector entities primarily through (1) the QRAC, whose  membership was comprised of individuals from academia, nonprofit  research organizations, private consultants, and nonprofit service  providers and advocacy organizations; and (2) the National Dialogue.  With regard to the QRAC, it met nine times during which it received  information from DHS leadership regarding the QHSR design, analysis,  and interim conclusions, and its members provided feedback and advice  to DHS. However, one QRAC respondent noted that the council’s  members were predominately consultants and not representatives of  industries affected by homeland security threats, such as critical  infrastructure sectors, which resulted in views that were not  representative of one of the most affected members of the nonfederal  homeland security community. This respondent stated that enhancing  participation of private sector representatives is important for the next  QHSR, as it would help DHS obtain views from entities that provide  homeland security and emergency responses services, such as one  corporation providing water to victims after Hurricane Katrina. According  to this stakeholder, private sector entities could help offer DHS views on,  for example, best practices for how to prepare for and respond to  homeland security events or technology enhancements for homeland  security.
With regard to the National Dialogue—one of the primary mechanisms  used for soliciting input from nonfederal stakeholders—17 stakeholders  who responded to our request for comments on the QHSR, as well as  NAPA, identified challenges. As an example of comments we received  from these 17 stakeholders, 1 federal stakeholder reported that the  National Dialogue did not appear to have significant impact on the QHSR  because in interagency meetings involving this stakeholder, information  from the National Dialogue was not discussed. In an additional example,  one QRAC member responded that the National Dialogue included a  small number of comments from the private sector and did not reflect the  significant number of stakeholders around the country with homeland  security responsibilities. This respondent stated that the National  Dialogue was an important exercise but was not an effective means for  obtaining representative views specifically of the private sector. Further,  as another example, one state and local association responded that in its  view, DHS’s conclusions on QHSR strategy had been reached prior to  initiation of the National Dialogue, making it appear to the association that  although DHS was soliciting its input, the department did not view the  association as playing a consultative role in the QHSR development. In  addition, NAPA reported that engaging nonfederal associations, such as  the National Association of Counties, did not necessarily equate to  reaching out to individual nonfederal entities, such as cities and counties.  Therefore, according to NAPA’s report, through the National Dialogue,  DHS notified approximately 1,000 contact members of nonfederal  associations in an effort to include a range of nonfederal homeland  security practitioners. Based on this outreach effort, NAPA’s report  recommended continuing efforts to gain significant buy-in from nonfederal  associations to ensure that DHS obtains access to the nonfederal  stakeholders it wishes to consult regarding the QHSR.
DHS faced challenges in obtaining nonfederal input during the QHSR  process for two reasons. First, convening state and local government  officials for consultation, especially from individual nonfederal  stakeholders, on the QHSR was a significant logistical challenge,  according to DHS officials. Because of this challenge, DHS opted to  consult with national associations that could represent the perspectives of  state, local, and tribal homeland security stakeholders. Second, according  to the Deputy Assistant Secretary for Policy (Strategic Plans) at DHS, the  Federal Advisory Committee Act (FACA), which establishes standards  and uniform procedures for the establishment, operation, administration,  and duration of advisory committees, affected how DHS was able to  consult with private sector stakeholders when developing the QHSR  report. Specifically, the Deputy Assistant Secretary noted that the  department was limited in its ability to consult with private sector groups  on an ongoing basis without forming additional FACA committees  specifically for conducting consultations on the QHSR. DHS was also  limited in its ability to seek feedback from established FACA committees  that had been convened for other purposes. The meeting schedules of  those committees did not align well with the QHSR study period, and  there were significant logistical challenges to scheduling additional  meetings of those groups to address QHSR. In addition, the Deputy  Assistant Secretary for Policy (Strategic Plans) stated that under FACA  DHS could not invite members of established FACA committees  convened for other purposes to join meetings of the QRAC for the  purpose of providing advice and feedback. One study group facilitator  commented that the FACA consideration significantly reduced the role  that nonfederal stakeholders played in the QHSR. According to this  respondent, addressing the FACA requirements and including appropriate  FACA-compliant groups with a broader range of academics and others  could have affected the outcome of the study group’s deliberations.  However, according to the Deputy Assistant Secretary, establishing new  FACA committees in addition to the QRAC, which DHS established as a  FACA-compliant committee specifically for QHSR consultations, was  prohibitively time consuming within the time frames DHS had for  conducting the 2010 QHSR.
Four respondents to our request for comments made suggestions for  alternative approaches for obtaining viewpoints of nonfederal  stakeholders in future QHSRs. For example, one study group facilitator  stated that state and local associations could put together a group of their  members to engage in the QHSR process and be part of the study  groups. In addition, the National Dialogue could have provided more  focused questions and provided to a broad group of state and local  experts questions on specific issues, such as housing disaster resiliency.  This approach could have allowed more state, local, private sector,  academic and nongovernmental organizations into the QHSR process,  according to the facilitator. DHS officials noted, though, that the National  Dialogue was not intended to address individual initiatives, as the QHSR  was intended to focus on broader homeland security issues. Further, one  local government association suggested that this association could put  together a crosscutting group of local officials who could discuss specific  issues, such as national preparedness. In addition to alternative  approaches for obtaining viewpoints of nonfederal stakeholders provided  by respondents outside of DHS, one DHS stakeholder responded they  held in-person or teleconferencing meetings with numerous interest  groups and associations, while another DHS stakeholder responded that  the component sent emails to its stakeholder groups soliciting the groups’  views on the QHSR. Additionally, in our prior work on the Federal  Emergency Management Agency’s (FEMA) process for updating the  National Response Framework, we identified examples of ways in which  FEMA involved nonfederal stakeholders in the process. For example,  FEMA posted a spreadsheet that included the comments made by  nonfederal stakeholders and the final disposition DHS assigned to each  of those comments to allow stakeholders to see how DHS did or did not  incorporate their comments. Further, FEMA had agency leaders appoint  advisory council members who represented a geographic and substantive  cross section of officials from the nonfederal community.
Given the significant role that state and local governments and the private  sector play in homeland security efforts, which is acknowledged by DHS  in the QHSR report, examining mechanisms, such as those proposed by  QHSR stakeholders or used by components, could help DHS include a  broader segment of these representatives during the QHSR process and  better position DHS to consider and incorporate, as appropriate,  nonfederal concerns and capabilities related to homeland security in the  next QHSR.
DHS identified stakeholders’ roles and responsibilities in the QHSR report  primarily by referencing other homeland security-related documents, such  as the National Response Framework and National Infrastructure  Protection Plan, that describe homeland security roles and  responsibilities. With regard to federal agencies, the QHSR report  described homeland security roles and responsibilities with brief  summaries of federal agencies’ leadership roles for coordinating  homeland security–related efforts. For example, the QHSR report listed  the Attorney General’s responsibilities as conducting criminal  investigations of terrorist acts or threats by individuals or groups,  collecting intelligence on terrorist activity within the United States, and  leading the Federal Bureau of Investigation, the Drug Enforcement  Administration, and the Bureau of Alcohol, Tobacco, Firearms and  Explosives in their respective areas of homeland security responsibilities.  With regard to nonfederal stakeholders’ roles and responsibilities, the  QHSR report provided summaries of roles and responsibilities, based on  these and other homeland security–related documents, such as  identifying that critical infrastructure owners and operators are  responsible for developing protective programs and measures to ensure  that systems and assets are secure from and resilient to threats.
Our prior work has shown that agencies that work together to define and  agree on their respective roles and responsibilities when implementing  federal strategies that cross agency boundaries can enhance the  effectiveness of interagency collaboration. In doing so, agencies clarify  who will do what, organize their joint and individual efforts, and facilitate  decision making. Further, our work on key characteristics for effective  national strategies identified, among others, one desirable characteristic  as defining the roles and responsibilities of the specific federal  departments, agencies, or offices involved and, where appropriate, the  different sectors, such as state, local, private, or international sectors.  Inclusion of stakeholders’ roles and responsibilities in a strategy that  crosses agency boundaries is useful to agencies and other stakeholders  in clarifying specific roles, particularly where there is overlap, and thus  enhancing both implementation and accountability. In addition, we have  reported that DHS needs to form effective and sustained partnerships  with a range of other entities, including other federal agencies, state and  local governments, and the private and nonprofit sectors. Successful  partnering involves collaborating and consulting with stakeholders to  develop and agree on goals, strategies, and roles to achieve a common  purpose.
In responding to our request for comments on the QHSR, 10 federal  stakeholders noted that the roles and responsibilities listed in the QHSR  report, as derived from other documents, such as the National Response  Framework and National Infrastructure Protection Plan, reflected their  homeland security missions and activities. For example, 1 federal  stakeholder responded that the roles and responsibilities listed in the  QHSR report were established in previous documents and were accurate,  and another federal stakeholder noted that the roles and responsibilities  listed in the QHSR report were derived from previously published material  for which the stakeholder had provided input. However, DHS and 10 other  respondents to our request for comments noted that the department could  strengthen its definition of homeland security roles and responsibilities in  the next QHSR by better reflecting the range of the stakeholders’ roles  and responsibilities. Specifically, in the QHSR report DHS identified the  need to better assess stakeholders’ homeland security roles and  responsibilities, noting that although the report was not intended to  describe stakeholders’ roles and responsibilities, the division of  operational roles and responsibilities among federal departments and  agencies for various homeland security goals and objectives emerged as  a major area requiring further study following the QHSR report. DHS  reported that an analysis of roles and responsibilities across the  homeland security missions would help resolve gaps or unnecessary  redundancies between departments and agencies going forward. Further,  10 stakeholders commented to us that the definitions of roles and  responsibilities in the QHSR report could be enhanced to better reflect the  range of homeland security stakeholders’ responsibilities. For example, 3  federal stakeholders reported that roles and responsibilities definitions in  the QHSR could be enhanced by, for example, recognizing the variety of  agency or administration-level responsibilities of the cabinet departments.  In particular, one of these federal stakeholders suggested that the next  QHSR may want to include a more detailed delineation of the roles and  responsibilities of departments to support the homeland security  enterprise by (1) reflecting the broad nature of responsibilities across a  broad spectrum of threats and (2) identifying readiness and resource  requirements to address the stated roles and responsibilities. The brief  narrative on roles and responsibilities in the 2010 QHSR report presented  a shortened version of the roles and responsibilities that the federal  stakeholder has in supporting the homeland security enterprise. Another  federal stakeholder noted that formalizing the process to elicit values and  judgments from individual agencies would help ensure adequate  representation of each agency’s role in the next version of the QHSR  report. The formalized process, according to the federal stakeholder,  would convene agency officials and facilitate a discussion, resulting in a  common understanding of how agency roles and responsibilities are  defined for executing the QHSR strategy.
In our December 2010 report on the extent to which the QHSR report  addressed reporting elements that the 9/11 Commission Act specified for  the report, we noted that DHS partially addressed two reporting elements  for the QHSR report related to roles and responsibilities for homeland  security stakeholders. These elements were for the QHSR report to  include a discussion of the status of (1) cooperation among federal  agencies in the effort to promote national security and (2) cooperation  between the federal government and state, local, and tribal governments  in preventing terrorist attacks and preparing for emergency response to  threats to national homeland security. With regard to the first element,  we reported that although the QHSR and BUR reports discussed  homeland security roles and responsibilities for federal agencies, they did  not discuss cooperation on homeland security efforts among federal  agencies other than DHS. We reported that while the QHSR discussion of  roles and responsibilities as found in other documents was helpful for  understanding which federal agencies lead particular homeland security  efforts, the QHSR report did not provide a description of how federal  agencies cooperate with one another in addressing homeland security  efforts. With regard to the second element, we reported that although the  QHSR and BUR reports provided descriptions of cooperation between  DHS and state, local, and tribal governments, they did not discuss the  status of cooperation between other federal agencies that have homeland  security responsibilities and state, local, and tribal governments. DHS  officials stated that DHS solicited comments from other federal  departments and state, local, and tribal governments on the role and  responsibility descriptions for each of these entities listed in the QHSR  report.
According to the Deputy Assistant Secretary for Policy (Strategic Plans)  at DHS, during the QHSR process the department did not attempt to  discuss the status of cooperation among other federal departments and  between other federal departments and state, local, and tribal  governments. DHS officials stated that the department viewed such a  discussion as outside its authority to conduct and that those discussions  were conducted in other venues, such as the National Infrastructure  Protection Plan and the National Response Framework. Because the  National Response Framework and the National Infrastructure Protection  Plan were completed during DHS’s launch of the QHSR, in 2008, use of  those definitions in the QHSR was appropriate, according to the official.  DHS did not obtain comments from all stakeholders on the definitions  listed in the QHSR report, but looked at stakeholder comments on roles  and responsibilities received during the National Infrastructure Protection  Plan and National Response Framework drafting processes. The  definitions listed in the QHSR report were also shared with DHS’s Office  of Intergovernmental Affairs, which solicited comments from stakeholders,  as necessary, based on any roles that may have changed since the  National Infrastructure Protection Plan and National Response  Framework were published, according to the official.
In its May 2010 report on the QHSR, the QRAC noted that the QHSR  report included a summary of roles and responsibilities of key  stakeholders that was derived from existing statutes, among other  documents. However, according to the QRAC report, the QHSR report  did not provide a mapping of these roles and responsibilities to the QHSR  missions and further work was required to deconflict and potentially  supplement existing homeland security stakeholder role and responsibility  policies and directives. According to the QRAC report, the QHSR report  was designed to create a shared vision of homeland security in order to  achieve unity of purpose; a vital next step was the delineation of key roles  and responsibilities for individual QHSR goals and objectives to generate  unity of effort. A comprehensive mapping of stakeholder roles and  responsibilities to QHSR missions, goals, and objectives was needed to  (1) enable assessment of the current state of cooperation and  coordination between all public and private sector stakeholder  communities; (2) identify potential gaps, conflicts, or both in current  policies and directives from an enterprise perspective; and (3) underpin  follow-on planning efforts. The QRAC recommended that DHS map goals  to objectives for each core QHSR mission and key stakeholder  communities to delineate the stakeholders’ respective roles and  responsibilities. In response to this recommendation, DHS plans to map  the existing QHSR mission goals and objectives to stakeholder roles and  responsibilities during the pre-execution year for the next QHSR, if  possible. This setup work on role and responsibility mapping would allow  for work at the end of the QHSR process to map roles and responsibilities  to final QHSR goals and objectives developed during the next QHSR.  Consistent with the QRAC’s recommendation and DHS’s planned actions,  by seeking to further define homeland security stakeholders’ roles and  responsibilities in the next QHSR, DHS could be better positioned to  identify, understand, and address any potential gaps in roles and  responsibilities or areas for additional or enhanced cooperation and  coordination.
section_title: DHS Plans to Implement a National Risk Assessment as Part of the Next QHSR
paragraphs: Through the QHSR, DHS identified various threats confronting homeland  security but did not conduct a risk assessment for the QHSR. In the 2010  QHSR report, DHS identified six threats confronting homeland security,  such as high-consequence weapons of mass destruction and illicit  trafficking and related transnational crime, as well as five global  challenges, including economic and financial instability and sophisticated  and broadly available technology. According to the QHSR report, these  threats and challenges were the backdrop against which DHS planned to  pursue its homeland security efforts. The threats and global challenges  listed in the QHSR report were developed through discussions with  federal national security officials and through reviews of intelligence  community materials, according to DHS officials.
Multiple DHS guidance documents emphasize the importance of  considering risk assessment information when engaging in strategic  decisions. For example, DHS’s Integrated Risk Management Framework  (IRMF), published in January 2009, calls for DHS to use risk assessments  to inform DHS-wide decision-making processes. Risk assessments,  which include assessing and analyzing risk in terms of threats,  vulnerabilities, and consequences of a potential homeland security  incident, are the foundation for developing alternative strategies for  managing risk, according to the IRMF. Similarly, the QHSR report  includes an objective for DHS to establish an approach for national-level  homeland security risk assessments, specifically calling for development  and implementation of a methodology to conduct national-level homeland  security risk assessments. Our prior work on federal strategic studies has  also found that establishing an analytic framework to assess risks is a key  aspect of developing a strategy to address national problems, such as  homeland security. Consistent with the IRMF, we define risk  assessment as a qualitative determination, a quantitative determination,  or both of the likelihood of an adverse event occurring and the severity, or  impact, of its consequences.
DHS has called for the use of national risk assessments for homeland  security but did not conduct such an assessment as part of the 2010  QHSR. DHS officials stated that at the time DHS conducted the QHSR,  DHS did not have a well-developed methodology or the analytical  resources to complete a national risk assessment that would include  likelihood and consequence assessments. The QHSR terms of reference,  which established the QHSR process, also stated that at the time the  QHSR was launched, the homeland security enterprise lacked a process  and a methodology for consistently and defensibly assessing risk at a  national level and using the results of such an assessment to drive  strategic prioritization and resource decisions.
In recognition of a need to develop a national risk assessment  methodology, the QHSR National Risk Assessment Study Group was  created as part of the QHSR process. In establishing the study group, the  QHSR Terms of Reference stated that assessing national risk was a  fundamental and critical element of an overall risk management process,  with the ultimate goal of improving the ability of decision makers to make  rational judgments about tradeoffs between courses of action to manage  homeland security risks. The QHSR National Risk Assessment Study  Group consulted with subject matter experts from the federal government,  academia, and the private sector and, in October 2009, produced the  Homeland Security National Risk Assessment (HSNRA) methodology,  which established a process for conducting a national risk assessment in  the future. According to DHS officials, because the HSNRA  methodology was developed as part of the QHSR process and finalized  as the QHSR report was being completed in late 2009, it was not  intended to be implemented during the 2010 QHSR.
The HSNRA is to use a methodology for assessing risk across a range of  hazards for use by DHS in its decisions on strategy and policy  development, planning priorities, resource allocation, and capability  requirements development. The HSNRA includes definitions or  descriptions of the scope of incidents it applies to, risk formula, and  likelihood and consequence (see table 2).
Outputs from the HSNRA calculations could be expressed in a number of  ways, such as plotting scenarios on a two-dimensional graph with  scenario frequency estimates on the x axis and scenario consequence  estimates on the y axis, as shown in figure 3.
In accordance with the QHSR goal of implementing a national risk  assessment and with issuance of Presidential Policy Directive 8, which  calls for risk analysis across a range of homeland security threats, DHS is  planning to conduct a national risk assessment as part of its next  QHSR. In determining how to conduct a national risk assessment, DHS  is considering various factors, such as how to incorporate and use an  assessment’s results, the time frames and costs for conducting an  assessment, and what alternatives exist to conducting a national  assessment.    Use of national risk assessment results. DHS officials stated that one  consideration in determining how to conduct a national risk  assessment is the manner in which the department would use the  results of such an assessment to inform the QHSR. Specifically, DHS  officials told us that a national risk assessment, such as the HSNRA,  should be one of multiple inputs considered in conducting the QHSR,  with other inputs including such factors as privacy and civil liberties  concerns, economic interests, and administration priorities. DHS’s risk  assessment guidance makes a similar point, stating that risk  information is usually one of multiple factors decision makers consider  and is not necessarily the sole factor influencing a decision. There  may be times when the strategy selected and implemented does not  optimally reduce risk, and decision makers should consider all factors  when selecting and implementing strategies. The National Research  Council of the National Academies also reported that risk analysis is  one input to decision making, designed to inform decisions but not  dictate outcomes. DHS officials noted that it would be important to  communicate this to its stakeholders, including Congress, the public,  and others, to manage any expectations that QHSR decisions would  be solely based on risk assessment results.    National risk assessment time frames and costs. According to DHS  officials, the HSNRA would require 12 months to complete and would  need to be completed before launching the next QHSR, since the  assessment would help frame how the QHSR missions are defined. If  the next QHSR is conducted during fiscal year 2013 and reported by  December 2013, as anticipated by DHS, the HSNRA would need to  be completed during fiscal year 2012 to help inform the QHSR,  according to DHS officials. With regard to financial costs, DHS  officials estimated that conducting the HSNRA over a 12-month period  would cost from $3 million to $6 million. DHS’s Deputy Assistant  Secretary for Policy (Strategic Plans) stated that the HSNRA is a  sound methodology that should be used as part of the next QHSR,  and officials within DHS’s unit responsible for developing the HSNRA,  the Office of Risk Management and Analysis, stated that the benefits  of having risk information available for input into developing the QHSR  are worth the costs.    National risk assessment alternatives. In order to identify risks and  inform mission areas for the next QHSR, DHS could consider  alternatives to conducting a national risk assessment, according to  DHS officials. These officials stated that one alternative approach  would involve using segments of the HSNRA process to help provide  risk information to department decision makers, such as eliciting  expert judgments and surveying nonfederal experts about  perspectives on the risks DHS should address. The officials stated  that this approach would not be as useful as a complete HSNRA  because a full HSNRA provides likelihood and consequence  estimates for various homeland security incident scenarios, which  offers a more complete picture of the risks DHS must address.  Another approach, according to the officials, would be to identify risks  through existing DHS analyses, such as the Homeland Security  Threat Assessment or the National Planning Scenarios. The officials  stated that identification of risks through these tools would also be  limited and would not be as effective as completing the HSNRA. For  example, the HSNRA includes likelihood estimates for scenarios,  which these other tools do not include, and therefore provides a more  complete picture of risk by addressing threats, likelihoods, and  consequences.
Consistent with DHS’s plans, a national risk assessment conducted in  advance of the next QHSR could assist DHS in developing QHSR  missions that target homeland security risks and could allow DHS to  demonstrate how it is reducing risk across multiple hazards.
section_title: DHS Developed BUR Monitoring Mechanisms and Measures but Could Strengthen Its Prioritization Efforts by Using Risk Information
subsections: section_title: DHS Prioritized Its BUR Initiatives but Could Benefit from Considering Risk Information in Future Efforts
paragraphs: DHS considered various factors in identifying high-priority BUR initiatives  for implementation in fiscal year 2012 but did not include risk information  as one of these factors. Through the BUR, DHS identified 43 initiatives  aligned with the QHSR mission areas to help strengthen DHS’s activities  and serve as mechanisms for implementing those mission areas.  According to DHS officials, the department could not implement all of  these initiatives in fiscal year 2012 because of, among other things,  resource constraints and organizational or legislative changes that would  need to be made to implement some of the initiatives. In identifying which  BUR initiatives to prioritize for implementation in fiscal year 2012, DHS  leadership considered (1) “importance,” that is, how soon the initiative  needed to be implemented; (2) “maturity,” that is, how soon the initiative  could be implemented; and (3) “priority,” that is, whether the initiative  enhanced secretarial or presidential priorities. Component leadership  officials, as subject matter experts, completed a survey instrument  indicating their assessment of each BUR initiative based on these criteria.  The results were then aggregated and presented to DHS’s Program  Review Board—which is the body that oversees DHS program reviews  and the budgeting process. With the Deputy Secretary’s leadership, the  Program Review Board evaluated the results of the survey and refined  the prioritization. The BUR initiative prioritization process resulted in the  Secretary and Deputy Secretary of Homeland Security ranking and  selecting 14 high-priority BUR initiatives to be implemented in fiscal year  2012, as shown in table 3.
We and DHS have called for the use of risk information in making  prioritization, resource, and investment decisions. For example, DHS’s  IRMF states that DHS is to use risk information to inform strategies,  processes, and decisions to enhance security and to work in a unified  manner to manage risks to the nation’s homeland security. The IRMF  states that one of its objectives is to use an integrated risk management  process to inform resource allocations on a departmentwide basis, which  is critical to balance resources across the set of DHS strategic objectives.  Likewise, our prior work has shown the importance of using risk  information to inform resource prioritization decisions. For example, our  risk management approach advises using risk information to inform  resource allocation decisions so that management can consider which  risks should be managed immediately and which risks can be deferred  and addressed at a later time.
According to DHS officials, using risk information as an input into DHS’s  prioritization of the initiatives was difficult for several reasons. For  example, the BUR initiatives were highly differentiated, making  comparisons based on risks the initiatives address impossible, according  to DHS officials. Some of the BUR initiatives focus on organizational  changes at DHS; others are extremely broad, addressing multiple and  overlapping risks; and others focus on specific risks. For example,  comprehensive immigration reform is a broad BUR initiative, addressing  broad illegal immigration risks, while promoting safeguards for access to  secure areas in critical facilities targets more specific risks. According to  the officials, the variance in how the initiatives were defined allowed DHS  to align initiatives with the QHSR strategy and consideration of such  variance, in addition to risks addressed by QHSR implementation  mechanisms, such as BUR initiatives, would be important in defining  implementation mechanisms and initiatives for future QHSRs. However,  DHS could not apply its existing risk assessment tools to evaluating and  prioritizing BUR initiatives for the 2010 QHSR.
For future QHSRs, DHS officials described several characteristics of  mechanisms for implementing QHSR missions that would enable risk  information to be used among prioritization criteria. First, the  implementation mechanisms or initiatives to be prioritized based on risk  information should be comparable in terms of the nature of the risks  addressed. For example, comparing mechanisms to address DHS  organizational changes that do not directly reduce homeland security  risks with mechanisms that are designed to directly prevent terrorism  risks would be an inappropriate comparison. Second, expected outcomes  of the mechanisms or initiatives should be defined so that the risks  reduced by the mechanisms can be estimated. For example, the BUR  initiatives do not indicate the degree to which investments will change  DHS’s security capabilities. Knowing the increase (or decrease) in  security capabilities associated with an implementation mechanism would  allow estimates of risks reduced, which could be compared in  prioritization efforts. Third, an implementation mechanism or initiatives  should have a “line of sight” directly between the DHS activities  associated with the mechanism and the risk reduced by those activities.  In other words, according to the officials, DHS operations need to be  closely aligned with identified risk reductions in order for risk reduction  calculations to be accurately achieved. For example, U.S. Border Patrol  efforts to stop illegal border crossings are closely aligned with reducing  risks of illegal immigration.
DHS officials stated that although existing DHS risk assessment tools  could not be used to systematically prioritize the BUR initiatives for the  2010 QHSR, there is utility in thinking qualitatively about risks addressed  by the initiatives when making future prioritization decisions. Risk  information should not be the sole input but should be considered along  with other criteria, according to Office of Risk Management and Analysis  officials and the Deputy Assistant Secretary for Policy (Strategic Plans).  DHS has various tools that could, with some limitations, provide risk  information for consideration when prioritizing implementation of QHSR  mission objectives, as shown in table 4. Two tools, the Risk Analysis  Process for Informed Decision-making (RAPID) and the methodology for  conducting the HSNRA, were created to provide risk information for  decision making across DHS mission areas. Risk analyses conducted  within DHS components could also provide risk information useful for  prioritizing QHSR implementation mechanisms, according to DHS  officials. The officials stated that at least five current risk assessments  used by DHS components could be useful for prioritizing QHSR  implementation efforts within the mission areas relevant to the risk  assessment tools.
DHS officials stated that there are benefits to considering risk information  in resource allocation decisions; however, DHS has not yet examined the  extent to which risk information could be used when implementing  subsequent QHSRs. Consideration of risk information could help  strengthen DHS’s prioritization of mechanisms for implementing the  QHSR, including determining which initiatives or programs should be  implemented in the short or longer term and the resources required for  implementation. Such information could also help the department to more  effectively make decisions about implementing initiatives and allocating  resources across initiatives that address different levels and types of  risks.
section_title: DHS Has Developed Plans and Scorecards for Managing and Monitoring Implementation of BUR Initiatives
paragraphs: DHS is managing and monitoring its implementation of BUR initiatives  primarily through its budget development and execution process, called  the Planning, Programming, Budgeting, and Execution (PPBE) process.  The objective of the PPBE process is to articulate DHS’s goals,  objectives, and priorities; align DHS programs with those goals; guide the  development of the department’s budget request; and set guidelines for  implementing the current budget. To manage implementation of the BUR  initiatives, beginning with the fiscal year 2012 budget request, DHS  officials told us that DHS developed implementation plans for each of the  43 BUR initiatives during the planning phase of the PPBE process (see  fig. 4). DHS assigned a directorate, component, or office to lead  departmentwide implementation efforts for each initiative, including  developing the implementation plans. According to DHS, each  implementation plan included what needs to be done to accomplish the  BUR initiative, what is currently being done to address identified  implementation problems, a description of stakeholders involved in the  implementation effort, and a discussion of next steps. DHS initiative leads  submitted BUR implementation plans to the department for review and  discussion during the fiscal year 2012 budget development process and  the fiscal years 2012-2016 budget review process, and these plans  served as a basis for components to develop their Resource Allocation  Plans (RAP)—components’ descriptions of funding needs for fiscal year  2012.
All DHS components execute  missions, spend resources, and  report on performance.
Key activities:  DHS is to  translate planning priorities into  5-year resource and performance  plan (the Future Years Homeland  Security Program), and allocate  limited resources to best meet  the prioritized needs.
Key activities: DHS is to  develop detailed budget estimates  of approved resource plans for  budget year justification and  presentation, and work with the  Office of Management and Budget  and Congress to get a budget  enacted.
Key activities:  DHS Chief Financial  Officer monitors accountability and  execution of budget authority and  reports results and makes  recommendations on realigning  resources. Funds are apportioned to  the directorates and components in  accordance with apportionment  guidelines.
Outputs:  Secretary’s  Resource Allocation Decisions,  DHS Future Years  Homeland Security Program.
Outputs: Budget request sent  to the Office of Management and  Budget and Congress’ briefing  and information sent to Congress.
Outputs:  Monthly Budget Execution  Reports, Midyear Review,  congressionally directed reports,  Annual Financial Report, and Annual  Performance Report.
DHS plans to implement the BUR initiatives primarily through  components’ existing programs and activities. For example, DHS plans to  implement the strengthen aviation security BUR initiative through its  existing aviation security programs, such as checking airline passengers  against watchlists and screening passengers at airports. Through the  PPBE process for fiscal year 2012, DHS requested additional funding for  select BUR initiatives, above base funding for programs and activities that  support those initiatives. For example, under the Improve Detention and  Removal Process BUR initiative, DHS requested about $222 million in  increased funding to support its existing Secure Communities program  and effort to rightsize detention bed space. In addition to increased  funding requested for select BUR initiatives, according to DHS, the  department planned to fund existing programs and activities that support  the other BUR initiatives through its base funding. For example, the  Domestic Nuclear Detection Office stated that it plans to fund its  programs that support the BUR initiative to increase efforts to detect and  counter nuclear and biological weapons and dangerous materials through  its base funding.
To monitor implementation of the BUR initiatives, DHS established  scorecards as part of its Integrated Planning Guidance—which DHS  developed during the planning phase of the fiscal year 2013 PPBE  process to provide guidance to DHS components for the programming  and budgeting phases. The scorecards depict the status of  implementing BUR initiatives, including, among other things, whether  DHS requested funding for BUR initiatives in fiscal year 2012 or plans to  request funding in future years. The scorecards also allow DHS to  periodically assess progress made on implementing individual BUR  initiatives and the status of BUR implementation as a whole. For those  BUR initiatives for which the department did not identify specific funding  needs in future years, DHS officials told us that they have discussions  with DHS components and directorates during midyear budget review  meetings to discuss progress made toward implementing BUR initiatives.  In addition, DHS officials told us that because the BUR initiatives reflect  existing DHS priorities, the initiatives are monitored through the Secretary  of Homeland Security’s discussions with component and directorate  leadership, such as discussions on progress being made on a particular  BUR initiative like strengthening aviation security.
section_title: DHS Has Undertaken Efforts to Link Its Performance Measures to QHSR Mission Areas
paragraphs: DHS has taken action to develop and strengthen its performance  measures, including linking them to QHSR missions and goals and  ensuring limited overlap among measures. While DHS has not developed  performance measures for all QHSR missions, goals, and objectives,  DHS has efforts under way to develop measures to address those  missions, goals, and objectives. Our prior work on key practices for  performance measurement has shown that measuring performance  allows organizations to track the progress they are making toward their  goals and gives managers critical information on which to base decisions  for improving their performance. We also have previously reported on  attributes of successful performance measures that include ensuring that  measures are linked to agencies’ missions and goals.
Since issuance of the QHSR report, DHS has undertaken efforts to  develop new performance measures and link its existing measures to the  QHSR missions and goals. These efforts included DHS providing  guidance to components that outlines how to assess QHSR missions,  goals, and objectives and achievement of QHSR outcomes. DHS also  provided components with performance measure development training  and formed working groups to discuss performance measurement best  practices. To support these efforts, in 2010, we provided technical  assistance to DHS and its components as they developed and revised  their performance measures to align with the strategic missions and goals  of the QHSR. Our feedback ranged from pointing out components’  limited use of outcome-oriented performance measures to assess the  results or effectiveness of programs to raising questions about the steps  taken by DHS or its components to ensure the reliability and verification  of performance data. While we offered advice on best practices for  performance measurement and developing outcome-oriented measures,  we did not suggest specific performance measures or targets or  recommend methodologies for collecting, analyzing, and reporting  performance measure data. Therefore, there was no expectation that we  and DHS reached agreement on the performance measures, and thus  decisions related to performance measures were fundamentally an  executive branch management responsibility. In response to this  feedback and its internal review efforts, DHS took action to develop and  revise its performance goals and measures to strengthen its ability to  assess its outcomes and progress in key mission areas. In DHS’s fiscal  years 2010-2012 Annual Performance Report, DHS identified 57 new  performance measures for fiscal year 2011, retained 28 measures from  the fiscal year 2010 measure set, and is in the process of refining the  methodologies for additional measures that the department plans to  implement in fiscal year 2012.
DHS’s actions to strengthen its performance measures have helped the  department link its measures to QHSR missions, goals, and objectives.  DHS has not yet developed performance measures for all of the QHSR  goals and objectives but has plans to do so. Specifically, DHS has  established new performance measures, or linked existing measures, to  13 of 14 QHSR goals, and to 3 of the 4 goals for the sixth category of  DHS activities—Providing Essential Support to National and Economic  Security. DHS reported these measures in its fiscal years 2010-2012  Annual Performance Report. At the time of issuance of that report, DHS  had not yet developed performance measures for QHSR Goal 2.3,  Disrupt and Dismantle Transnational Criminal Organizations, or one of  the goals for its sixth category of activities—Provide Specialized National  Defense Capabilities. However, since then, DHS officials told us that the  department has developed performance measures for these goals and  plans to publish them in its budget justification to Congress upon approval  of the measures by DHS leadership and the Office of Management and  Budget. Further, within QHSR Goal 4.2, Promote Cybersecurity  Knowledge and Innovation, DHS has not yet developed measures for two  of the three objectives—foster a dynamic workforce and invest in  innovative technologies, techniques, and procedures. DHS officials told  us that the department is collaborating with the Office of Personnel  Management on a multiyear effort to identify competencies and more  accurately gauge workforce needs for cybersecurity professionals and is  working to develop a measure related to innovative technologies that  have been developed and deployed.
section_title: Conclusions
paragraphs: Homeland security includes a vast range of mission areas—from  preventing terrorism to securing U.S. borders, safeguarding cyberspace,  and ensuring resilience to disasters. It also involves a wide variety of  stakeholders and partners, including federal departments and agencies;  state, local, and tribal governments; and nongovernmental entities,  including the private sector. Given the scope and magnitude of the  homeland security enterprise, it is important for the federal government to  set clear goals, objectives, and priorities for securing the United States  and making resource allocation decisions. DHS’s 2010 QHSR—the  department’s first quadrennial review—was a massive undertaking to  review the nation’s homeland security strategy and identify homeland  security missions and organizational objectives. It involved the input of  numerous stakeholders with homeland security roles and responsibilities,  including other federal agencies, state and local government entities, and  academics. DHS plans to initiate its next QHSR in fiscal year 2013 and to  report on that review’s results in fiscal year 2014. In conducting this next  review, DHS could leverage lessons learned from the 2010 QHSR to  strengthen its planning and risk management efforts. Specifically, given  the array of federal and nonfederal stakeholders involved in implementing  homeland security missions, building more time for obtaining  stakeholders’ feedback and input and examining additional mechanisms  to obtain nonfederal stakeholders’ input could strengthen DHS’s planning  and management of stakeholder consultations and better position it to  obtain, review, and incorporate, as appropriate, stakeholders’ feedback.
Risk assessment in the homeland security realm is an evolving field,  although DHS has developed methodologies, human capital, and  departmental policies for integrating risk information into DHS decision- making processes. Such information can help decision makers identify  and assess homeland security threats and vulnerabilities facing the nation  and evaluate strategies for mitigating or addressing those threats and  vulnerabilities. Using existing risk assessment tools could assist DHS in  prioritizing QHSR implementation mechanisms. Specifically, examining  the extent to which risk information could be used to help prioritize  implementation mechanisms for the next QHSR could help DHS  determine how to incorporate and use such information to strengthen  prioritization and resource allocation decisions.
section_title: Recommendations for Executive Action
paragraphs: To strengthen DHS’s planning, management, and execution of the next  QHSR, we recommend that the DHS Assistant Secretary for Policy take  the following three actions:    Provide more time for consulting with stakeholders during the QHSR  process to help ensure that stakeholders are provided the time  needed to review QHSR documents and provide input into the review,  and build this time into the department’s project planning for the next  QHSR.    Examine additional mechanisms for obtaining input from nonfederal  stakeholders during the QHSR process, such as whether panels of  state, local, and tribal government officials or components’ existing  advisory or other groups could be useful, and use them for obtaining  nonfederal stakeholders’ input, as appropriate, during the next QHSR.    Examine the extent to which risk information could be used as one  input to prioritize QHSR implementing mechanisms, including  reviewing the extent to which the mechanisms could include  characteristics, such as defined outcomes, to allow for comparisons of  the risks addressed by each mechanism. To the extent that DHS  determines that risk information could be used, consider such  information as one input into the decision-making process for  prioritizing the QHSR implementation mechanisms.
section_title: Agency Comments and Our Evaluation
paragraphs: We requested comments on a draft of this report from DHS. On  September 12, 2011, DHS provided written comments, which are  reprinted in appendix III. DHS concurred with our three recommendations  and described actions planned to address them. With regard to our first  recommendation that DHS provide more time for consulting with  stakeholders during the QHSR process and to build this time into the  department’s project planning for the next QHSR, DHS stated that it  would endeavor to incorporate increased opportunities and time for  stakeholder engagement during the next QHSR. Regarding our second  recommendation that DHS examine additional mechanisms for obtaining  input from nonfederal stakeholders during the QHSR process and use  them for obtaining nonfederal stakeholders’ input, DHS stated that it will  examine using panels of state, local, and tribal government officials and  existing advisory groups to obtain input. With regard to our third  recommendation that DHS examine the extent to which risk information  could be used as one input into prioritizing QHSR implementing  mechanisms and to consider such information, if appropriate, when  prioritizing QHSR implementation, DHS stated that it intends to conduct  risk analysis specific to the QHSR in advance of the next review. DHS  stated that it plans to consider the results of such analysis, along with  other factors, as an input into decision making related to QHSR  implementation. DHS also provided technical comments, which we  incorporated as appropriate.
We also requested comments on a draft of this report from the  Departments of Agriculture, Defense, Health and Human Services, State,  the Treasury, and Justice and the Office of the Director of National  Intelligence. The Department of Defense provided technical comments  which we incorporated as appropriate. In e-mails received from  departmental liaisons on September 7, 2011, the Departments of  Agriculture, State, and Justice indicated that they had no comments on  the report. In e-mails received on September 7, 2011, from the  Department of the Treasury’s Director for Emergency Programs and the  Department of Health and Human Service’s Office of the Assistant  Secretary for Legislation, both departments indicated that they had no  comments on the report. In an e-mail received on September 9, 2011,  from a departmental liaison, the Office of the Director of National  Intelligence indicated that it had no comments on the report.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 20 days from the  report date. At that time, we will send copies to the Secretaries of  Agriculture, Defense, Health and Human Services, Homeland Security,  State, and the Treasury; the Attorney General; the Director of National  Intelligence; and selected congressional committees. The report also will  be available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions, please contact me at (202) 512- 9627 or maurerd@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix IV.
section_title: Appendix I: DHS Strategic Documents and the National Security Strategy Align with the QHSR
paragraphs: The Department of Homeland Security’s (DHS) strategic documents,  such as component strategic plans and budget requests, align with the  Quadrennial Homeland Security Review (QHSR) missions. The May 2010  National Security Strategy (NSS) also identifies strategic elements related  to homeland security that are identified in the QHSR report, such as  similar listings of homeland security threats.
Each of the DHS strategic documents we reviewed includes language  explicitly aligning at least some aspects of the strategy with the QHSR  report, as shown in table 5. According to DHS officials, DHS does not  have an explicit policy that strategic documents, such as component  strategic plans, be consistent with the missions, goals, and objectives  listed in the QHSR report. However, such consistency is expected by  DHS senior management, according to the officials. We also identified 17  references to homeland security within the NSS that relate to DHS  responsibilities. While not explicitly linked to the QHSR report in the NSS  document, each of the 17 statements link to aspects of the QHSR report,  such as homeland security threats identified or specific QHSR goals or  objectives.
section_title: Appendix II: Scope and Methodology
paragraphs: The objectives for this report were to evaluate the extent to which the  Department of Homeland Security (DHS) (1) consulted with stakeholders  in developing the Quadrennial Homeland Security Review (QHSR)  strategy; (2) conducted a national risk assessment to develop the QHSR;  and (3) developed priorities, plans, monitoring mechanisms, and  performance measures for implementing the QHSR and Bottom-Up  Review (BUR) initiatives.
To address our objectives, we analyzed DHS documents related to the  QHSR, BUR, and budget development processes, including the QHSR  report, BUR report, fiscal year 2012 budget request, and Fiscal Years  2012-2016 Future Years Homeland Security Program. We identified  criteria for evaluating these processes by analyzing our prior reports on  key characteristics of effective national strategies, key practices for  effective interagency collaboration, strategic planning, performance  measurement, and standards for internal control, among others. For a  listing of these prior reports, see the related products listed at the end of  this report. Based on these reports, we identified those key practices and  characteristics applicable to quadrennial reviews, like the QHSR. The key  practices we identified were involving stakeholders in defining QHSR  missions and outcomes; defining homeland security problems and  assessing risks; including homeland security strategy goals, subordinate  objectives, activities, and performance measures; including resources,  investments, and risk management; including organizational roles,  responsibilities, and coordination across the homeland security  enterprise; and establishing a DHS process for managing implementation  of BUR initiatives. We vetted the key practices with our subject matter  experts—staff with legal and methodological expertise and experience  analyzing the Quadrennial Defense Review—and provided them to DHS  officials for review and incorporated their comments as appropriate. As  we developed our report, we grouped these key practices into three  areas—stakeholder involvement, risk assessment, and implementation  processes for the QHSR and BUR initiatives.
Because respondents volunteered information about their views on the QHSR, we do not  know the extent to which other officials within the same organizations shared these views. provided insights into stakeholder perspectives on how QHSR  stakeholder consultations were conducted and how they could be  improved. Further, we reviewed reports on the QHSR by the National  Academy of Public Administration (NAPA) and the QRAC, both of which  were based upon each organization’s collaboration experiences with DHS  in developing the QHSR report. During the QHSR, NAPA partnered with  DHS to conduct three National Dialogues, which allowed any member of  the public to review draft QHSR material and provide online suggestions  for the QHSR. According to the QRAC’s report, the QRAC served as a  forum in which committee members, who were nonfederal  representatives, shared independent advice with DHS on the QHSR  process. We compared DHS’s stakeholder consultation efforts to our prior  work on effective practices for collaboration and consultation. For  example, based on a key practice in federal agency collaboration, we  analyzed the extent to which DHS worked with stakeholders to establish  agency roles and responsibilities when developing the QHSR.
To determine the extent to which DHS conducted a national risk  assessment to develop the QHSR, we analyzed risk analysis–related  documents produced as part of the QHSR process, such as DHS risk  assessment tools, and interviewed DHS officials responsible for  developing risk analyses for use at DHS. We compared DHS’s risk  assessment process in the QHSR to our prior work on key characteristics  for risk assessment as well as DHS risk analysis guidance documents.  For example, we reviewed our previous reports on key practices in risk  management, including risk assessment approaches, and compared  them to DHS’s effort to develop a national risk assessment methodology.  In addition, we reviewed DHS guidance for use of risk assessment  information and compared the guidance with DHS’s QHSR risk  assessment process.
To determine the extent to which DHS developed priorities,  implementation plans, monitoring mechanisms, and performance  measures, we analyzed DHS’s BUR implementation priorities and plans,  such as DHS’s fiscal year 2012 budget request; monitoring mechanisms,  such as BUR initiative scorecards; and DHS’s performance measures.  We also interviewed DHS officials responsible for managing and  monitoring the implementation of the BUR initiatives. We compared  DHS’s processes for prioritizing, monitoring, and measuring  implementation efforts to our prior work on key practices for risk  management and implementation and monitoring of strategic initiatives.  For example, we identified practices in our past reports and DHS  guidance for using risk information in resource prioritization decisions and  compared DHS’s efforts to prioritize and implement the QHSR strategy  with those practices. We also compared DHS’s strategic-level  performance measures for fiscal year 2011 to our criteria on key  attributes of successful performance measures. Because DHS focused  on aligning its performance measures with QHSR missions, we selected  three key attributes of successful performance measures that were most  relevant—linkage, core program activity, and limited overlap. In applying  the attributes, we analyzed documentation, such as the QHSR report and  DHS’s fiscal years 2010-2012 Annual Performance Report. We also  interviewed DHS officials who are involved in overseeing the  development and reporting of DHS performance measures.
We conducted this performance audit from January 2011 through  September 2011 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix III: Comments from the Department of Homeland Security
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Rebecca Gambler, Assistant  Director, and Ben Atwater, Analyst-in-Charge, managed this assignment.  Jean Orland and Janay Sam made significant contributions to this work.  Michele Fejfar assisted with design and methodology, and Tracey King  provided legal support. Labony Chakraborty, Jessica Orr, and Robert  Robinson assisted with report preparation.
section_title: Related GAO Products
paragraphs: Defense Transportation: Additional Information Is Needed for DOD’s  Mobility Capabilities and Requirements Study 2016 to Fully Address All of  Its Study Objectives. GAO-11-82R. Washington, D.C.: December 8, 2010.
Department of Homeland Security: Actions Taken Toward Management  Integration, but a Comprehensive Strategy Is Still Needed. GAO-10-131.  Washington, D.C.: November 20, 2009.
Transportation Security: Comprehensive Risk Assessments and Stronger  Internal Controls Needed to Help Inform TSA Resource Allocation.  GAO-09-492. Washington, D.C.: March 27, 2009.
Quadrennial Defense Review: Future Reviews Could Benefit from  Improved Department of Defense Analyses and Changes to Legislative  Requirements. GAO-07-709. Washington, D.C.: September 14, 2007.
Risk Management: Further Refinements Needed to Assess Risks and  Prioritize Protective Measures at Ports and Other Critical Infrastructure.  GAO-06-91. Washington, D.C.: December 15, 2005.
Results-Oriented Government: Practices That Can Help Enhance and  Sustain Collaboration among Federal Agencies. GAO-06-15. Washington,  D.C.: October 21, 2005.
Results-Oriented Government: Improvements to DHS’s Planning Process  Would Enhance Usefulness and Accountability. GAO-05-300.  Washington, D.C.: March 31, 2005.
Combating Terrorism: Evaluation of Selected Characteristics in National  Strategies Related to Terrorism. GAO-04-408T. Washington, D.C.:  February 3, 2004.
Tax Administration: IRS Needs to Further Refine Its Tax Filing Season  Performance Measures. GAO-03-143. Washington, D.C.: November 22,  2002.
Homeland Security: Proposal for Cabinet Agency Has Merit, But  Implementation Will be Pivotal to Success. GAO-02-886T. Washington,  D.C.: June 25, 2002.
Standards for Internal Control in the Federal Government.  GAO/AIMD-00-21.3.1. Washington, D.C.: November 1999.
Executive Guide: Effectively Implementing the Government Performance  and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1996.